Maybe Imu can make you ageless if you share your Soul with Imu?
So, in order to become a Gorosei, you have to give Imu part of your Soul. Same thing with whatever Gunko's position is as Imu's attendant: this fits well, since it would be 6 people, a number associated with Demonology.
If the Celestial Dragons knew reincarnation was real, it would make a lot of sense for most of them to not want to share their Soul with Imu, and be trapped in the same age/body forever.
I think this is why Mars had the dialogue he did when he looked at Vegapunk's Brain in Punk Records:
Mars almost seemed... whimsical, towards the idea of death, here.
Imu only being able to possess people who have shared their Soul with Imu makes a lot of sense.
It is possible that Imu can take over anyone who has a Mark of the Abyss, too, though.