It's basically people with their own agendas.
Gorosei being above "PK-level" (which is where they rightly should be for the 800 -year reign to make any sense whatsoever) is something that destroys the agendas of the Yonko stans and many admiral stans.
If they are that strong, that'd mean they could've wiped out the Yonko had they wanted to. Which goes against the Yonko stand agenda of Yonko being strong enough to rival the World Government.
If they are that strong, that'd mean the admirals aren't the big baddies for the WG, which admiral stands get insecure about due to it being a point in their Admiral > Yonko agenda. Even though Gorosei > Admiral has 0 connection with Admiral > Yonko.
But yea, Gorosei are them dogs. I have been rereading pre-TS lately, up to end of Water 7 saga so far, and Oda's subtle writing with the building up Gorosei as the big antagonists of the world govrenment has been nothing short of top notch, tbh.