What if he’s just the ultimate troll? What if he was requested to test Luffy in the East Blue by Shanks? What if he deliberately got himself sent to Impel Down because he knew Luffy would break in? He was able to clash with Mihawk, successfully protect Luffy and come out unharmed in Marineford.
All of his loyal followers like the Impel Down inmates are being subjected to his Conquerors Haki, speaking of which he was able to get in and fool the guards into thinking he had no DF. Potentially that’s his advanced Voice of All Things mind controlling people.
And we all know the Marines failed to capture Buggy when the Warlord System was disbanded. Not even the admirals could stop him: