Yes. Exactly this!
And please don't get me wrong if I was criticising monolithic projects.
It was exactly _right_ to get pixelfed up fast and working. That was most important to incubate fediverse but I think now it's time for generic servers (and diverse clients).
About “modular”:
I am working on https://github.com/redaktor/widgets-preview/ [using dojo.io, similar to React]
and all widgets take AP objects natively and then you just need to decide by type(s).
But modularity is hard and takes time :(