This is honestly a really good question. And it kind of made more sense when I thought they were all likely from the Void Century together, just maybe using different techniques than Imu's to stay alive. But, since they're sharing Imu's power, and possibly all come from different time periods, it does make less sense to me. It seems pretty likely that once you get the power, your age just stops, and you don't slowly age.
I think the answer will likely be more "metaphorical" than "practical"? Because, practically speaking, yeah, it would make more sense to have your strongest warriors stay at the peak of their strength in their 20s or 30s.
But, the Gorosei aren't JUST warriors. They're also managers and administrators. And those positions take time to learn and grow into. It's possible it just takes that long to rise in the Celestial Dragon ranks and stick out enough that Imu notices you.
....Which, if that's the case, Ju Peter MIGHT actually be the most impressive out of all the Gorosei. You have to wonder, what exactly did he DO to make Imu notice him at such a relatively young age? Ju Peter might have been some sort of prodigy.
It's also possible that Imu just likes bossing around "old people". If Imu does turn out to be some sort of immortal eternal child, like I personally think is possible, then it's possible they just enjoy making older people do what they say. It's just fun for them, to make these people who otherwise would be bossing THEM around for being a kid, do whatever they want.
Or maybe Imu just values their age and experience?
It's also possible that it has something to do with "dreams" or "ambition"?
Maybe these older people have had their dreams crushed by the passage of time, which makes them easier to control?
Or maybe they have more to lose, since they have children and grandchildren of their own, so they'll work harder at their jobs for Imu?
Or maybe Imu NEEDS the bitterness and regret that sometimes comes with old age in order to get a foothold in to grant them power? Enhancing their darker emotions with Imu's own power?
Or maybe it has something to do with the fear of death, that becomes more real as you get older? Again, it might make them easier to control or manipulate.
So, I don't know. There's probably a reason. But, it's hard to say what with any certainty, right now. We still don't know a lot about the Gorosei or Imu.