I think it's more likely Xebec abandoned them, or more specifically Kaido. Kaido never mentions Xebec as one of the men he respects, plus he was the youngest and likely most impressionable member of the Rocks Pirates. I can see a scenario where Kaido once looked up to Xebec only to realize it was misplaced. Maybe Kaido views preservation of life as a form of cowardice because Xebec in the end was a coward who'd rather run to live his days rather than die with glory.