Don't get me wrong, I think the approach Signal takes is very reasonable and probably a reason for its success.
It is good practical/*usable* security and users who need more privacy will face a lot of hurdles in operational security most of their communication partners probably won't take on - so it usually becomes a moot point.
Large adoption base is the biggest privacy preserving factor - and Signal is doing good there *because* of its tradeoffs.
Everyone of our communication partners who do have our phone number and still use WhatsApp, etc. will give a part of our social graph to those companies, no matter how good *our* operational security is.