fediDB.org is a valuable resource for the fediverse in many ways

I'm aware of the lack of opt-out and robots.txt support

In the near future, #fediDB will respect robots.txt rules, and provide a manual opt-out feature to remove your instance on demand