Gorosei being strong or not both seem pretty much bad. If a gorosei is strong enough to subdue luffy... I don't see why they are having problems taking care of yonkos and pirates in general. It seems easy to manage once tip gybs are gone, they have to clean up small fries and pirates lose confidence too.
If they are just trash guys... I have no idea why they are in the series, we have seen absolutely nothing about them or whatever they did... No different than politicians of one piece world too.. And given how the series is... I doubt it's good to be like that. They are not seen as actual threat where politics doesn't really matter but just fights