Would Imu even care if those bums ran out of food?
Well, they've kept them around this long for some reason.
So, it's possible Imu and the Gorosei really care about them. Maybe they're all descendants of old friends that they promised to take care of? Maybe they all remind them of old friends long lost, and they want to keep them around for sentimental reasons. Which is why they get spoiled so much. They may legitimately care about them.
...Or they're all going to be used like human sacrifices in order to do...something?
Maybe if you throw enough people into one of those holes in the ocean, it stops the water levels from rising? Maybe they all get to live like kings, because eventually they're all going to get sacrificed to the whirlpools?
Or maybe Imu's special powers actually come from the Celestial Dragons? Maybe Imu's siphoning off a small amount of their life forces in order to get the power to empower the Gorosei? Think like Goku's Spirit Bomb, only more parasitic. Maybe it doesn't matter if they're strong, as long as there's a bunch of them that Imu can steal energy from?
Or maybe Imu has been keeping this group of people childish for a reason? Like, once they die, all their unfulfilled wishes and dreams turn into Devil Fruits? Maybe the Celestial Dragons have been purposely isolated in order to keep their dreams as "pure" and "childlike" as possible so that they create more creative Devil Fruits once they die?
So, chances are that Imu probably doesn't want them to die....at least until the time is right.