Theory:
Buggy used his grand fleet to find Captain John's treasure after Luffy handed over that map thing to him in Impel Down.
Think it's deemed to be the second most sought after treasure after the One Piece. If it's something very valuable to World Government or a serious weapon then it would justify the Yonkou status.
His crew are Level 6 inmates plus some other big hitters.
He is a former crew mate of the Pirate King and talks to a current Yonkou casually, while others would shit their pants. Even with haki I don't think any swordsman in the world can beat him, it would just be a stalemate.
In fact I think his crew are probably more competent and formidable than the Beast Pirates! While a lot of people are dismissing it as a gag and Oda is being Oda...
On paper it makes sense.