Signal is free and open-source. It cannot be denied that basically everything, including minor details like usernames, is end-to-end encrypted and kept secure. The Signal protocol has been proven to be secure by many independent experts and thus it is mathematically impossible for Signal to gain access to your sensitive information (except for your phone number, obviously).
A phone number alone just won’t do much.
Signal’s server is open-source. Of course, they could do something else in secret, but the openness of the client (here’s the client) is enough to verify that E2EE exists.
Your phone number alone just doesn’t give any real insight: you can derive that the person behind it prefers to communicate in private and that they’re probably alive, but that’s about it. Also, I don’t think Signal can get your name without a government to look it up. That does happen sometimes, it’s just that nothing importmant ever comes out of it.