Them being out on the seas without protection especially in yonko territory would be the dumbest thing for them to do. It doesn’t matter how strong the gorosei are their importance in the WG means that if they’re to travel the seas protection if for no other reason than to make sure the secret of their powers remain secret.
Its not really when they have been fully shown to handle themselves and quite literally exist to protect someone else (Imu). We are ignoring the pure fact that they are here because said person. Why wouldn't they be fought when Imu is fought? Makes no sense lol.
Their powers are barely being shown either, Saturn has zero named moves and we have no idea how his regen, eye shit or floor immobile shit even works.
Occams razor man, Fleet Admiral antagonist will have Admiral and Zodiac VA subordinates with him. WG antagonist will have Gorosei/God Knight. subordinates with him.
Hell, when the arc first began, Kizaru was the dedicated antagonist with VA subordinates in classic fashion. This arc is only unique on the sense that Kizaru is being fought by Luffy, and even Saturn's role has now escalated beyond that fact since the other 4 are here.
They also need to deliver the mother flame, why do they "need" to chase Luffy to Elbaf lol.