It’s time to accept Admirals are actually strongest force in the government next to Figarland
Like it was explicitly said in the first place.
But again, it's logic: if Gorosei were that strong, Xebec, Roger, WB, the Great Era of pirates, etc..., would have never survived. Them + Admirals + Garp + Garling and so on would have destroyed the pirates.
Instead, even they told us they were worried, idk, to anger Kaido

.