It completely destroys it lol.
As things stand Imu is a glorified Charloss with a big gun.
There’s a reason we never got Luffy vs Spandam or Luffy vs Orochi.
Imu sama will still be the final antagonist even if Luffy must fight someone else to bring Imu sama down.
Imu sama has narrative significance that Spandam and Charloss did not.
Stopping Imu sama will be crucial in a way that stopping neither of them was.
Imu is the embodiment of 800 years of tyranny and oppression; that significance trumps all powerscaling considerations. The powerscaling serves the story not vice versa. Oda will find a way to make it work out.
The Gorosei already exist to serve as Imu sama's bodyguards. Oda has leeway to introduce arbitrary other characters to fight in Imu sama's place if he wants to.
But that won't change the fact that Imu sama is very much still the big bad that must be taken down to liberate the world.
We might even have a case where Imu sama uses their weapon to try and destroy the entire world or something, and Luffy has to stop them. I don't know how Oda will handle it, but however he does, Imu sama will be
the Big Bad.