You have to take character statements with a grain of salt. Garp saying in his prime he would do a better galaxy impact, which isn't something we've actually seen(his Haki Dc in Egghead is like, probably better than anyone we've seen) is as reliable as Crydo saying no one in the world can beat him. They're just exaggerating their overall strength, and Garp is just glorifying how he was in his prime, but no one actually views him as lesser or weaker. Kuzan specifically knows how strong he was in his prime and isn't like "wow you've seriously declined etc".
At least with Rayleigh and Gaban you can make an argument that they retired, but even then, they just have straight up better or the same feats as old men than they do in their primes. It's pretty obvious that Oda is going to be super liberal about age and characters reaching their primes, given the fact that essentially every top tier except the Strawhats is past "Their prime" by irl standards. Kaido, Sakazuki, Dragon etc are like 60 and are in their absolute prime. It's even more obvious when you remember Imcel, and the Gorosei exist and they're all like centuries old, but they aren't really portrayed as getting much weaker, even though they're literally modeled after irl old politicians.
Haki and the Devil fruits are essentially equalizers here. As long as Garp keeps his Haki, he's pretty much the same in PL, except maybe his stamina is worse or whatever, but it's nothing crazy.