In my opinion, to paraphrase Ratatouille...
"Anyone can CoC. Not everyone CAN become a great Haki-man. But, a great Haki-man CAN come from anywhere."
It's more about willpower and determination than lineage. It's about having a dream you work towards no matter what. Lineage CAN help. But, it's not the sole determining factor for getting it. Just like raw physical strength isn't the sole factor.
Kizaru is strong but doesn't have Conquerors, because he isn't trying to fulfill his own dream. Meanwhile, Chinjao really isn't that strong, relatively speaking, but he DOES have Conquerors because he has a dream he wants to fulfill.
It's more about having the determination to make your dreams into reality that allows you to use Conqueror's. as far as I can tell. Maybe I'm wrong, but that's what it seems like to me, right now. Anyone, with sufficient effort, can get Conqueror's. It's just that very few people in the world have that level of dedication and willpower.