With the gorosei and holy knights we have our answer, why the government could withstand the yonkos.
Imu >> gorosei >= yonko = holy knights leader >= holy knights = fleet admiral >= admirals.
I think the gorosei are around prime whitebeard /shanks level. Meaning wb and shanks would rank probably somewhere between the 5. The same applies to the holy knights, the strongest knights are probably around yonko level, while the weakest is somewhere at admiral level. The leader of thr holy knights is probably on par with the strongest gorosei. I believe this is the reason why garp and roger stopped rocks, they knew they could not defeat the gorosei.
That's why, government > pirates. Till joy boy comes back.