Well, Whitebeard said about the Impel Down Prisoners that were with Buggy and that aren't even the strongest (since Blackbeard took the strongest), that he doesn't want to deal with them at that point in time because they look strong. Whitebeard just gives respect