I don't think that it would be a very wise move to attack the giant village. I don't think Broggy and Dory are the strongest warrior. There is most likely there the man power of one or two yonko crew.
Sorry to say but if it's a case of all five Gorosei chasing them, Shanks stands no chance either. Only way this ends well is if they have some kind of agreement with Shanks to leave Elbaf alone, which may have something to do with the Figarland theory
But yeah, the WG is never going to let their biggest weapons (other than Imu's) fall into enemy hands easily