Email or username:

Password:

Forgot your password?
Mastodon Engineering

We are excited to launch a new project that will help small and medium-sized fediverse servers and their users have better access to search and discovery through the use of pluggable Fediverse Discovery Providers, supported by a grant from @EC_NGI. See our new dedicated website for details:

fediscovery.org/

40 comments
:PUA: Shlee fucked around and

@MastodonEngineering @EC_NGI I for one welcome our new universal search pluggable providers!

Vyr โ˜ข๏ธโ˜ฃ๏ธ

@EC_NGI @MastodonEngineering current Fedi search is inadequate for most users on many platforms, and independently, pluggable providers for anything seem like a major step forward for Mastodon. definitely going to be watching this.

ChiefGyk3D

@MastodonEngineering @EC_NGI this is pretty cool, and means it will be even easier to use Mastodon and such over time. As a small server for just me and a few friends this will be fantastic. I am already pretty well federated but this will make it by far easier in the future.

ChiefGyk3D

@schizanon allow us to eventually have better search and discovery for smaller instances on the fediverse such as my privately run Mastodon server.

๐Ÿ„๐ŸŒˆ๐ŸŽฎ๐Ÿ’ป๐Ÿšฒ๐Ÿฅ“๐ŸŽƒ๐Ÿ’€๐Ÿด๐Ÿ›ป๐Ÿ‡บ๐Ÿ‡ธ

@chiefgyk3d so they say, but how does that work? As I understand it, your instance can delegate search to another instance instead of having to manage the index yourself? What if the results come from instances you don't federate with, or have fediblocked? What if there's CSAM or something in the results?

Jasdemi

@MastodonEngineering @EC_NGI This is gonna be great for single user instances.

C.W. Smith

@jasdemi

@MastodonEngineering @EC_NGI

It will definitely make running a single user instance a bit easier from scratch.

Concidering I need to think up how I want to do community features on a few sites it maybe help a lot there as well.

C.W. Smith

@schizanon

Looks like it will allow for a plugin ecosystem for search and search providers in a federated system. Making it easier to find people through search as opposed to going through instances and searching separately.

It's optional as well so.

Andrรฉ Menrath

@wysteriary @MastodonEngineering @EC_NGI There are key differences between relays and a generic search provider like proposed here. I think pointing those out is crucial. I hope that a flow graph of a draft, maybe including a direct comparison, will be available soon. I discussed this quite some time with the Mastodon guys and #FOSDEM and it was not easy for me to get the point.

hybrid havoc

@wysteriary Feels like relays will still serve a separate purpose, but as they mention in the post that these Discovery Providers may eventually serve other purposes as well, it could be possible.

With proper federated search and discovery features I would feel less need for relays.

rexi

@hybridhavoc @wysteriary
I have a few security and privacy questions, too

Andy Piper

@wysteriary @MastodonEngineering @EC_NGI it may end up somewhat similar to a relay - we are in definition phase. It is not the intent and it may have other requirements.

katzenberger

@MastodonEngineering

Despite this being promoted as supporting multiple "discovery providers", the text says ยปThis will specify how an instance will โ€œfeedโ€ content to a discovery provider to index. And how a discovery provider can be queried to actually search and discover content.ยซ

Nothing is being said about why there would ever be a need for a more than a few big providers, once it is established that the big ones "work". As usual, centralization is looming here, by a few corps "generously" offering high-speed "discovery" for "free".

Also, nothing is said about providers sharing results of the indexing, to enable fediverse-wide search - instead, the proposal almost inevitably points to few providers that everybody needs to feed if they want posts to be found.

"Hope" will not change that hence just saying ยปWe want to specify how a discovery provider works and interfaces with a Fediverse server instance. We hope to inspire several competing implementations of the specification.ยซ does not make much sense.

Nothing is said about who outside of the Fediverse gets to query a provider, or to process the data that has been extracted from a fediverse server feed - and for which purposes. It's even expressly mentioned that ยปThese โ€œprovidersโ€ might serve other purposes than just search and discoveryยซ

ยปThe protocols and implementations will respect user privacyยซ, but the text only offers "shoulds":

-

ยปOur reference implementation will respect this setting and only ingest content from creators who opted in to discovery in the first place.ยซ (note the telling words "respect" and "ingest" here, it does not even seem consensus that providers are "fed" by Fediverse servers).


-

ยปAll other information a discovery provider gathers should be anonymous.ยซ Should?


-

ยปFediverse Discovery Providers should only ever index content clearly marked as โ€œpublicโ€.ยซ Should?


-

ยปFediverse Discovery Providers should honor these settings and only index user data and content from users who have opted-into that.ยซ Should?

Another red flag: ยปThis means we might not always be able to incorporate all the feedback we get into the very first draft of everything we publish. But rest assured that we are committed to continue working on this even after this first project has ended, so we will be able to make adjustments later on.ยซ

More privacy after the first implementation is out? And no: With Mastodon gGmbH being the driver of this, and with the current wording, we can not "rest assured".

@EC_NGI

@MastodonEngineering

Despite this being promoted as supporting multiple "discovery providers", the text says ยปThis will specify how an instance will โ€œfeedโ€ content to a discovery provider to index. And how a discovery provider can be queried to actually search and discover content.ยซ

Nothing is being said about why there would ever be a need for a more than a few big providers, once it is established that the big ones "work". As usual, centralization is looming here, by a few corps "generously" offering...

ophiocephalic ๐Ÿ

@katzenberger
"Nothing is said about who outside of the Fediverse gets to query a provider, or to process the data that has been extracted from a fediverse server feed - and for which purposes."

The fact that the referenced passage links to a blog post about third-party data collection and algorithmic contraband media detection might provide a clue

katzenberger

@ophiocephalic

Another clue there: ยปWhy external providers?

As it stands, scaling a Mastodon instance is hard, as the operators need to form a moderation team with 24/7 availability and good reaction times, learn about all the specific vectors for abuse and common issues, design guidelines and more.ยซ

Their problem is "scaling". Self-inflicted pain from running behemoth servers that are counter to the core idea of the Fediverse, and from deliberately tilting the Mastodon "joining" process so that their own behemoth keeps growing.

@ophiocephalic

Another clue there: ยปWhy external providers?

As it stands, scaling a Mastodon instance is hard, as the operators need to form a moderation team with 24/7 availability and good reaction times, learn about all the specific vectors for abuse and common issues, design guidelines and more.ยซ

ophiocephalic ๐Ÿ

@katzenberger
Exactly. The "trust and safety" proposal in the older post is effectively for mastodon.social. So it's rather curious that it's being pointed at in a different proposal which is seemingly about improving search and discovery for smaller instances

ophiocephalic ๐Ÿ

@MastodonEngineering @EC_NGI
"These providers might serve other purposes than just search and discovery" - This passage links to @renchap blog post on "trust & safety". Does this mean this will be a mechanism for backend third-party data collection and algorithmic surveillance?

Chris Wood

@MastodonEngineering @EC_NGI@ec.social-network.europa.eu perfect! I think this will really help new users find their crowd - especially those who use social media to share their specific skills, products or services and are looking to network with a broad range of people.

Project Insanity

@kletterli das w#re doch vllt was fรผr uns zum aufsetzen und testen?

Alex Holst

@MastodonEngineering @EC_NGI

Protecting our trans and bipoc homies from evil accounts on mastodon dot social when?

amy

@MastodonEngineering@mastodon.social @EC_NGI@ec.social-network.europa.eu Cool idea but how would you implement content searching and also keep it anonymous? If all you want to do is publish trends, then sure, but that doesn't seem to have as much value add as content searching..?

Chris (Master of Potate) ๐Ÿฅ”

@scottytrees @MastodonEngineering Better search. If an admin connects their instance to a search provider, the posts from their instance will be searchable by all other connected instances. But this will not happen for a while since this is just an announcement that this will be developed.

Henry

@MastodonEngineering @EC_NGI I think itโ€™s best for account holders from another server can creat a delegate account on a popular server to allow proactive exposures of the account, not like passive waiting someone to follow your account then your existing will be recognised by that server.

โธธ commander โ–‘ nova โธธ :~$

@MastodonEngineering @EC_NGI I wonder if managed hosts will be implementing this, would like to see it through toot.io managed hosting, at least

abeorch

@MastodonEngineering

@EC_NGI @thibaultmol

Are you considering whether licences could be added / tagged to content to make it clear content can be used? - Is that relevant?

Astro

@MastodonEngineering @EC_NGI Where do I sign up to become notified once there is a spec regarding ascension from Relay to Provider?

Mastodon Engineering

@astro I am not sure I know what you refer to when using โ€œascension", but we will post here once we have a public repository with specifications.

Astro

@MastodonEngineering
Stoked! If I had free time I'd think #FediBuzz has a unique advantage for this "graduation" from Relay to Provider.

LucileDT is looking for work!

@MastodonEngineering @EC_NGI so cool! And will the moderation tools get some work too, to help people manage their communities?

Go Up