@TechConnectify So for it to be transparent, the algorithm, and it's training data should be transparent.
You also need to decide what signals you are going to include in your model. Here federation throws a wrench in things, as one server doesn't have immediate access to a lot of data on users from other servers.
Likes and boosts are fairly public, so you could probably use those as one bit of signal. Some sentiment analysis on the text could be a signal.
@TechConnectify Things like mutes/blocks are negative signals that could be used, but those are only available within a given server, so your servers filter could only use mutes and blocks by other users on your server as a form of input. I suppose that there might be enough data on larger servers for this to be reasonable.
Anyhow, it's an interesting problem, but I imagine it would take quite an effort to build something that would be useful.