If I had to put a money down prediction that people could hold against me once it's proven wrong, it would be that he didn't exclusively lose to them but also to some of his own crew defecting, because his crew straight up all betrayed him when the going got tough and Xebec can only do so much by himself.
I'm almost entirely expecting WB to join Roger and Garp in the take down because he'll dislike Rocks going postal on slave kids or something.
That's my simpleton prediction, you can laugh at me when it's wrong.