Yeah no shit the Gorosei didn't start shit, they'd have to be real fucking stupid to try. The Gorosei were scared of dying Whitebeard and Prime Shanks from allying. They'd be crapping their trousers if they had to fight what Shanks could bring. Keep in mind that if pre-Wano Kaido heard about this fight happening over the WG either killing or capturing Luffy, he would join in, too, as shown in Marineford (another moment two Yonkou meeting scared the crap out of the Gorosei).