CoC is the mandatory "chosen ones" "built different" archetype that some cultures love so much.
But, philosophically speaking, it means the 2nd. King managed to fight off Zoro. Some Conqueror users can be beaten by non conquerors. the haki itself, in the basic form, doesn't mean much against someone genuinely stronger than you.
But, as Rayleigh said, "people who have made a name for themselves, USUALLY have it".
It's the philosophical principle that "some people are meant for more". Now, personally you can agree or disagree with that, but here you can just accept this is the norm for Oda's PoV and move on.