This is the problem, we've come to associate masculinity with merely the physical qualities of someone, there are plenty of weak men with chest hair and beard.
Your mentality is what truly matters, the body is merely a vessel to express the spirit.
A true man is not attached to worldly things, he's not controlled by desires or any emotions, he doesn't own alot of things beyond the bare necessities and he doesn't depend on anything or anyone, that's a true man imo.
Someone who can't be bought, controlled, blackmailed or frightened, if you are subject to any of the above you're not truly there yet.