Tbh, I used to think of this too...
But Kaido confirmed a couple of chapters ago, that Xebec wasn't the kinda guy to fuck around with.
Like, didn't Kaido put Xebec along with Shanks, WB, Roger, and Oden as guys that
could fuck his ass?
EDIT: Since his crew was full of unruly people who wanted to kill each other, I'm guessing Xebec was the kinda guy that would settle these "disputes" by personally beating them up. So he personally kept everyone
in line, so to speak.. But with him gone (ALA Roger and Garp), the crew dismantled from those who survived.