Are you guys even reading the manga lol? Yes, Haki users can harm Holy knights and Gorosei, but they cannot harm Imu, we literally saw it
As opposed to regular regen merchants, who can be defeated
This isn´t rocket science, lol... Kaido is a different situation, because he never was invincible? Oda was comfortable showing him nearly dying to Oden before his arc fight, and him getting injured by the scabbards is another example, which is in clear contrast of Imu taking attacks from 6 of the top pirates/marines of an era.