Imu's Giant Frozen Hat probably doesn't belong to Joyboy. We saw Joyboy talking to Emet in the flashback and he looks around the size of a "normal" human. (Chapter 1122). Which raises a question of who that Giant Hat actually belongs to. It's possible that's actually Nika's Hat from the First World, but frankly we still don't know. If you look in the latest chapter 1160, there actually is a big guy (possibly a buccanner?) wearing a straw hat. I doubt that it's the same hat as the one on God Valley, but it's hard to say for sure.
I suppose Joyboy could have the power to shrink himself. Possibly using the Nika/Gum-Gum Fruit to shrink himself down. Basically, the opposite way Luffy usually uses it. But, I don't know why he'd do that while talking to Emet. It would kind of make sense for Joyboy to grow or shrink so he could talk to anyone he wants as "equals" at the same height, but in the only scene we see him, he's small. Who knows, though?
But, if Joyboy even has a drop of Giant blood in him, that could explain how he's so long lived. Even if he doesn't have enough Giant blood in him to make him Giant himself, he could still have slower aging than a "normal" human. He might only age half as fast as a "normal" human. Not the same slower speed as the full-blooded Giants, but slower than a normal human. Making him appear about 50, while still being 100 years old, or something.
We also have to remember, Joyboy grew up in the Ancient Kingdom. They were a highly advanced society. Just like how we're living longer in the real world, due to advances in medicine, it's possible the people of the Ancient Kingdom lived longer and looked younger, because of their advanced medicine. You'd be surprised what the right combination of vitamins can do. Joyboy might be "made of rubber"...because he's had some work done.
But, whatever way it happens, I doubt we'll see Old Man Joyboy. He'll probably look as old as Garp, at the oldest. Unless that's the tragedy of it all. Joyboy didn't fail because his plan was flawed. Joyboy failed because he grew too old to actually see it through. Joyboy waited to long. That'd be a interesting story. But, I'm not sure if it's one Oda wants to tell. Especially since it might hit a little too close to home, with his own health problems. Chances are, even if we see a Old Man Joyboy, he'll still be pretty vital up until the moment of his death.