No, the emphasis was just on Xebec this flashback. But nevertheless:
- According to the narrator, it was the era of Xebec, Garp and Roger
- Roger and Xebec were fierce rivals
- Garp apparently was most interested in rivaling Roger
- Captain of Rayleigh, who even in old age went toe to toe with Kizaru and made a Yonko retreat
- Random no name Roger pirates were already strong enough to waste guys like Kyo, who was said to be Shiki's rival
With Rayleigh easily wasting Sommers, Garling would've met the same fate against Roger.
Is it still fair to give Xebed the benefit of the doubt of being the strongest by virtue of the feats he displayed? Sure. But he isn't on a different level.