It might raise questions. How did they become Gorosei? Why don't they have Haki like Skanks and daddy Skanks?
On the assumption that Imu chose them (as opposed to them being ancient demons), I think it's pretty obvious what the deal is.
Imu is a sigma male and can't make powerful friends like alpha Figarlands. So he chose 5 intelligent guys who worship him unquestioningly, and made them powerful by giving them fruits, immortality surgery etc. much like Blackbeard's crew.
The have strong haki, but not Shanks tier. We need that so Imu looks like a big step up by comparison. It makes sense because they have the conqueror's will to actually rule the world, but still they're ultimately subservient followers.