It isn't. You have no standard to logically defend that six is too much but four, as we saw in the Roger Pirates (yes, Shanks included, doesn't matter he's a kid because he'd still have that spirit), is feasible.
...So? It's like saying the top people in their field can't cooperate and be friends only because they're the ones in a few millions —and surprise, it happens all the time! "One in a few millions" is still a significantly higher number than the twelve alive conquerors we know so far no matter how you put it.
Your "the actually number is 2" would be somewhat an argument (which it isn't) if Oden was using them instead of being their close friend and like family to Whitebeard; which he didn't. They got along perfectly well and that's the point, plus how you randomly distinguish Oden "using" Whitebeard and Roger from, I don't know, Franky "using" Luffy because he would need a crew to rule his ship in order to travel the world and achieve his dream. Also, still can't argument what's the factual distinction between two conquerors being perfectly capable to work together as the best friends in the world but six can't in spite of being as close as them and willing to die for each other. What actual argument do you have to support that Zoro's case or Rayleigh case can't happen that many times?
So again, Zoro, a conqueror, is willing to die for Luffy. Chinjao, a conqueror, went from wanting to kill him to sailing under his flag in like a day. Rayleigh, a conqueror, was crying as his partner said goodbye. Whitebeard, a conqueror who didn't trust other conquerors, ended up seeing Oden as his brother and wanting to make Ace the Pirate King (while Ace, a conqueror, gave his life for his father).
If all the Straw Hats were to be conquerors they would still follow Luffy because they are good people, because they are like a family, and because Luffy's greatest power is to make friends. Simple as that, there's no difference between a conqueror like Zoro subordinating to Luffy and another one, two, three... doing the same, because their friendship and Luffy's own charisma is above that —which is something Rocks and his crew lacked, hence why they failed.