He was at the top of fandom rankings in PL for most of the time. Admiral fans couldnt slander him because that would make their favs look pathetic. Yonko fans couldnt slander him because he was their poster boy (MF is probably where the Yonko vs Admiral debate began). Anti sword bros wanted to use WB title to disqualify/demerit Mihawk and save Shanks from WSS title. For the longest time PK was held as the highest level and WB being his equal, the one who sat before the throne but not on it even though he could.
Rayleigh was also similar, he was Luffys master with CoC, hyped by Garp, Sanji fans wanted to have parallels, anti sword bros ranked Ray above Mihawk etc.
These two guys were the most generally respected top tiers in the story and held onto that position for too long. The fan wars raged and new characters were introduced so the slander began. Neither WB nor Ray had a downplay moment in the flashbacks when read with context yet people just want to pull down those who are above.