I know shonen laws work the same way as video game laws, that once a character reaches a proverbial level that they stay there and don't go down...
But it simply makes no sense for Imu to be so strong, physically. Unless he non-stop sparrs and trains, which is doubtful, not regularly exercising fighting ability causes them to get sloppy. And he probably hasn't fought anyone in centuries.
At least figures like Kaido and Big Mom could theoretically go out and raid places.