What's wrong with the bolded part? There's no indication that he fought any of the Supernova who came to BM's territory, and apparently nobody else has been crazy enough to attack the Yonko in a long time. Who has Katakuri fought? At least King has to defend his position and took part in Wano's invasion.
And like
@HA001 said, people respected Katakuri until his fake perfect facade was shattered. They respected the lie that was the perfect older brother.
The Supernova question the authority of the Yonko, are they close to Yonko level? The Revolutionaries question the authority of the WG, are they close in power to the WG?
YC3 Vista respected Mihawk, does that effect their power levels in your mind?