The gorosei didn't give a shit about that.
And you think the agents showed up in Wano at whose request? Lol Just accept that the Goroseis aren't all that man...
They shit their pants out of fear of Kaidou's fury, they wanted wano in their possession but they never got it, they would try to "acquire wano" while Kaidou was at war with the entire alliance, that is, they need to use cowardly tactics, afraid of causing Kaidou's fury, they were like dogs crying to Orochi for weapons, they shit themselves at the mention of Whitebeard and Shanks meeting and there must be a lot more L of them, I mean look at how Saturn failed to kill the child Kuma, blackmailed by Doflamingo.
Don't step on the gas with Saturn which possibly has a trick, the guy got FUCKED by Kuma using basic COA. LOL
Stabbed by Bonney who didn't even use Basic Haki.