The only objects that can't be referenced through that /objects collection are the ones that are not addressed to either the shared inbox of the server, or the ActivityPub public namespace. (ie, objects generated by private Activities - addressed only to specific actors' inboxes)
I'm still not 100% sure this is the way I want to keep things, but for now it serves me well, and lowers the complexity of managing objects' IRIs.
@mariusor they do have unique URIs, but when they aren't in any collection, nothing links to them from anywhere, so they're kinda just floating there in the ether. Knowing that URI becomes the only way to discover them.