If the Gorosei are weak, how did the World Government last 800 years in charge, with no marine coup ? And how will they be proper antagonists in the face of Luffy and his allies ?
Unless Imu pulls a Madara and goes 1 vs All, the Gorosei are gonna be powerful fighters.