The Gorosei were afraid to invade Wano after the Raid, and it sure as FUCK isn't because of the Samurai, right?
So, I look at it like this.
Nerona Imu > Pluton > Gorosei
Gorosei knew from past experiences that they couldn't handle Pluton themselves. I'm guessing that Pluton was ordered to guard Wano.
Imu is decidedly above Pluton, though. Now that Imu is active, the WG has a force stronger than Pluton, allowing them to invade.
Pluton guarding the country and being above the Gorosei in power is pretty much the only satisfying explanation as to why the world government hasn't invaded the country since the void century.
They aren't scared of wano.
They're scared of pluton.