@brook if I may add my two cents: the part about unauthenticated API access breaking the website is, as far as I know (corrections welcome), new starting with version 4.

In Mastodon 4, Eugen / Mastodon gGmbH, in what I can only call a since-ongoing episode of "I know what's best and only I", decided that pre-rendered pages (i.e. a post on the instance is browsed to and the instance renders a static page showing the post, with no client-side API access happeneing) were obsolete, and replaced all such pages with dynamically rendered pages.

Besides breaking a whole bunch of features if I recall correctly, that also introduced this very problem.