Well, harsh but in a certain way it's true.
If I were the world government, I would shit myself if I were to fight Joyboy and his Red hair's level crew plus the future world strongest swordsman, plus an alliance of natural born monsters like Minks, Samurais, Kuja, maybe Giants and other fighting tribes, maybe the remnant of some Yonkous crews and several new world's crews, the Revolutionary army and finally one or two ancient weapons.
Luffy's destiny always has been to turn the whole world upside down, what do you think a mere Yonkoy represents to him (after an appropriate growth after 25 years of serialization)?