@tauli A problem that is probably not impossible to solve.
Without wanting to go into details here, because there are probably smarter minds involved with federation who could come up with a better strategy, this is a simple idea to approach this:
- instances are often specific to topics, e.g. space, biology etc, similar to how the StackExchange network already works.
@tauli Instances with related topics could follow each other to also search their index (or replicate the remote content in a local search index) to provide useful results from other instances.
This allows to keep the overall index small enough, because you don't have to index everything but only instances relevant to your's focus.