Xebec went after Kaidou—but actually, Kaidou invaded Hachinosu and defeated some of Xebec’s men. Xebec wanted to meet him, and as a precaution, he sent Whitebeard, the strongest member of his crew, to recruit 13-year-old Kaidou.
After that, and once Kaidou joined, the crew became unstoppable and invincible. That doesn’t seem like a small feat, and it doesn’t seem like Xebec would want a loser among them.