I think it'd be stupid if you can't detect an invisible man with CoO, given its strong enough.
They still exist in physical form, they can move and interact with objects around them. With haki, you don't just sense their physical forms, but also their willpowers. Its how you gauge the strength of an opponent simply by his haki.
Besides, I think you're all overrating it a bit. Just throw some shit on the guy and it becomes easy to defeat him.