Assuming Imu isn't an Okama, it can only really be a 50/50 thing, male or female. And after the 5 old Gorosei farts, I hope Oda gives us a Queen of the World
But to your question, I think it's partly fueled by the flower room (flowers = associated with women), plus the theory of Imu reversed being Umi (Japanese for sea), and being Luffy's mother (like Terra with Zoro and Sora(Sky) for Sanji).
But if this is about Lily from Alabasta and people from hundreds of years ago, I guess Imu won't be Luffy's mother and that theory flies out of the window