Rocks wasn't cancelling out the attacks: he was talking them and still regen them. Obv. not all of them: he manage to stop some of them, as Harald.
Roger and Garp couldn't go beyond his defence and deal actual long standing damange to him.
On the other hand, as said, 17 YO Loki, in 1vs1, was able to make Harald return to his sences and wasn't even that damaged.
After the got the fruit, the fight was quite one sided.
But I say even more: id It Is as you say, Loki would be even more incredibile since Roger and Garp couldn't go beyond Rocks defence while Loki did more than once.