Multiple factors.
- Pissger needed Garp's help
[1]
- It's heavily implied Imu intervened at God Valley
- Some of the Elders were seen during God Valley
- Rayleigh, Gaban, Garling were in their prime
- WB, Meme and Shiki weren't loyal to Xebec
- Kaido was fodder back then
- Rest of Rocks are canceled out by immortal HKs
Xebec getting extreme diffed by a combined effort of Roger, Garp and maybe a third party is a fairly plausible scenario and what happened most likely.