IMO,
the same way Enel started the surviving games with the intent of taking with him to the moon the winner,
so Imu decided to take with him the winners of a similar game into the new post-Ancient Kingdom world he was to make.
And the current Gorosei are the only survivors from the Ancient Kingdom that oppposed Imu.
Much like Enel with Nami, Imu did keep his promise and took the Gorosei with him into the Void Century (maybe taking them far enough into the sky to spare them from the world flood), except there was a catch. Imu only saved the Gorosei's bodies, not their actual souls/personalities. Instead, like Moria and his zombies, Imu animated said bodies with nightmare/demons he has control over, turning what were once the greatest warriors of the Ancient Kingdom in his own bodyguard slaves.