Didn't say he's just attack power, but even a scrub like DR-era Hajrudin can counter a 22 million pound weight from Machvise. Giants are almost unmatched in terms of raw physical power. Add an insane devil fruit + Ragnir + ACoC on top of that, and Shanks/Gaban just didn't have that level of destructive power.
But Kaido showed that in a pure 1 on 1 your endurance is tantamount, and your CoC is even more important.
The giants know of the power of Shanks' Haki.