Because he still managed to be seen as a big enough threat for Garp and Roger to put him down. He can still be a Buggy type whilst being really strong too. He doesn't need to be exactly like Buggy but just have his charisma, cunning and ingenuity especially with resourceful bullshitting he excels at.
Buggy is still considered a Yonko despite not being remotely close in strength to one. He's literally fodder and still managed to become a Yonko but he still garnered respect from powerhouses like Crocodile and Mihawk enough even though they refuse to admit it. People can have their feats, their hype, their strength exaggerated and misunderstood.
There are other ways to impress people even without pure strength, although Kaido would be impressed most by strength so I will give you that and thus what I'm suggesting is a bigger stretch then but still. It's also far more interesting than Xebec just being another generic unstoppable, "evil" powerhouse too, especially when he had such ABSOLUTE monsters already in strength, attitude, reputation and such willingly under his command too.
Xebec probably was that guy, he probably was that inhumanly strong but to me thats such cliche, generic, predictable, dull writing especially if he just got everyone to join him via strength and thus intimidation alone. I don't even need Killingham to send me to sleep again with such a yawn inducing yarn.
It makes it far more impressive if Xebec was not the strongest in the crew but still commanded respect, authority over them and kept them in line. I doubt there will be as much of an extreme power gap like Buggy vs Mihawk and Crocodile but I still hope he's not the strongest in the entire crew.
But at the same time, Garp and Roger had to teamup to take him down supposedly, so fuck it. Most predictable/generic scenario, lets go. I'll contain my hype.