I could see it being one of two things.
One, Imu might have some sort of Ancient Disease. We know there's some old diseases going around Mariejois, since Ginny caught Sapphire Scale there. It's possible that the reason why Saturn was experimenting on people in the first place was to try and find a way to cure Imu. Maybe spending time in the Flower Garden is the only thing holding Imu's disease back. And maybe Imu exerting himself is causing the disease to progress rapidly.
Or the second option, it might have something to do with the "terraforming" of this World. We know that there have been Three "Worlds" in the past. That implies that the World has been transformed before, likely due to different people becoming the "True God" of this World. If Imu lived during the Void Century, that means they lived in the Second World, before it was transformed into the current Third World we know about today. Since "Mu's World is still Incomplete", it's possible that Imu's body, which was "designed" to live in the Second World cannot handle the "atmosphere" of the Third World. The Flower Garden may have a environment that's similar enough to the Second World's environment that Imu can best survive there. It's a similar situation to "Nausicaa of the Valley of the Wind" (the manga version), where the humans were all altered to survive the pollution of their world, but cannot breathe the purified air once it's cleaned up. Imu's body may have never been "converted" to be compatible with the Third World's new environment.
https://en.wikipedia.org/wiki/Nausicaä_of_the_Valley_of_the_Wind_(manga)
Although, it is pretty likely that Imu does also have a time limit. Since that seems to be the case for Luffy in Gear 5, and even Loki in his Nidhogg form. Powerful Devil Fruits just take it out of you. So, that could also be a factor, but maybe not quite this early on in the battle.
It could be something else, but those are the two options that are coming to my mind. We'll see.
