Yep, he didn't only keep them all at bay but also Shiryu. It's clear that Shiryu was comparable to Magellan but even he was afraid of fighting Magellan. No wonders, top level paramecia + physical stats (after all, he was briefly injured when fighting Ivankov). That was Pre-TS. Now it wouldn't even wonder me if his Haki was very skilled as well.
Generally, all the Pre-TS powerhouses like the Warlords, Ace, Yami Blackbeard, Enel and so on are way stronger than the community thinks. When it comes to Ace, his best feats can easily rival those of the strongest commanders. And these feats happened when the manga reached +400 chapters. I think his novel feats/Ace manga feats are not that farfetched anymore.