There is a misconception that observation Haki makes invisibility useless. But not true, I mean even if we assume that Zoro for example knows about the power, having to compensate for the ability not to see the opponent demands a high level of Haki used for a long period. You screw up with Shiryu and you're dead.