The whole purpose of Navy is to be the outside face of WG and act according to what WG desires.
With admirals around you don't really expect any pirate to lay a hand on nobles let along Gorosei.
So, Saturn coming only for seraphim makes sense.
But thing is he is dealing with nut cases - Luffy and zoro who doesn't give a shit about anyone.
Despite being weak and knowing that an admiral will show up, luffy still punched a CD.
Now they are strong.
We all know it - the moment Luffy got pissed he won't even think twice to come after Saturn - and at this moment even Kizaru won't be able to do anything - Because of other SHs present like Zoro occupying him.
Saturn himself needs to be powerful