Problem is that Xebec isn't being framed as a bad guy in GV, at least. His only aim here is to save his wife and kids. So, that already puts in question "Rocks was so bad Roger and Garp had to team up against him."
If you want to count out the obvious answer of Domi Reversi, maybe Rocks, furious after the death of his wife, starts murdering every celestial dragon (and their slaves) in sight. Oda really have to emphasize the "also attacking slaves" part, though, or Roger and Garp will look like abhorrent CD dickriders.