Imu has been repeatedly implied to be one of the original twenty founders of the World Government
And according to Doflamingo, the twenty founders of the World Government were all kings
Of course a king by default is male
Also, assuming Imu is the final villain, which is looking most likely, Oda would naturally have Luffy beat down a man rather than a woman.