Literally nothing about the World Government's power structure makes sense if the Gorosei are not powerful.
They thought the most hyped old men, in a series full of badass old dudes, needed to be babysat by Admirals. Ugh.
Works well, too.
Marines = stronger than every individual Yonkou crew, but unable to fight two Yonkou crews at once
-3 Admirals
Yonkou = Could beat the Marines in alliances, but too divided and untrustworthy/untrusting to make alliances work
-4 Yonkou
World Government = Possesses an Ancient Weapon (most likely), and is ruled by "The King of the World" Imu. Unknown force, but by ruling the world, the strongest force
-5 Gorosei
As long as the Marines and WG are allied, the pirates are fucked. Which is why Sword exists, to weaken the WG for the final battles.