Buggy was bad mouthing Shanks in Marineford and picking on him. The Impel Down prisoners saw this as confirmation of Buggy being godlike, to be able to just pick up a fight with Shanks like that. Have you seen Shanks caring?
Again, you can't just assume Mihawk's power level based on him not carrying about public perception. Because apparently, Crocodile doesn't care either, even though in Marineford, he wasn't even willing to team up with Doflamingo on equal grounds. So him accepting this public perception of being an underling to a literal clown, means, that there is some grander scheme. So there is no reason to jump on that bandwagon.
Can you explain how Crocodile, who would've only accepted Doflamingo as a subordinate, not however as a partner, is ok with being perceived as an underling of a fodder clown?