50s is literally prime age wtf
Conditioning matters.
You can't be in your prime when you're out of shape.
The fact Shanks had to remind Gaban how long it had actually been should be taken as reflecting the easy-going times Gaban enjoyed just living there.
I'm not suggesting he hadn't fought at all, even if just for fun, but it couldn't have been much.
I don't think we'll see Prime Gaban or much of the Roger crew until we get the definitive Roger flashback, which I believe IS coming, but personally believe won't happen until Lodestar.