You guys don´t know what you´re talking about.
Joyboy can cancel out Imu´s powers with his haki, but his haki can´t actually injure Imu. That´s why the Gorosei didn´t care about Luffy getting acoc, but they sent someone to kill him before he awakened his devil fruit/
It´s also why Rocks based his plan on getting 2 devil fruits, it´s because his ancestor, Davy Jones had a special DF to fight Imu. Not everything has to be directly stated, Oda literally showed you Imu being indifferent to multiple ACOC users