Yeah it doesn't....."the only one's who can fight with me" ahem......
Well it's simply not true. Akainu can definitely hang with Kaido. If Luffy can fight Kaido then so can Kizaru, Aokiji and Fujitora. Probably Greenbull as well.
Garp (who fought Rocks and Roger) and Sengoku who fought Shiki (and presumably Whitebeard) certainly could fight Kaido when they were in their prime. Kong is indicated to be even above them. Rayleigh is indicated to have been WSS during his prime (same coat as Mihawk) and should have been able to fight Kaido.
Imu-sama is an unknown, as are the Gorosei. The boss of CP-0 is an unknown quantity. Dragon should by hype also be able to fight Kaido.
We know for a fact Big Mom being able to fight Kaido. Mihawk stalemating Shanks means he could fight Kaido as well. Blackbeard can fight Kaido, nullify his extreme dragon toughness and deliver terrifying gura gura-punches.
Regarding who can beat Kaido 1vs1 that's a different question altogether. But being able to fight him for 1 hour, 5 hours or 1 day all of the above definitely qualify and if paired together they can absolutely take him down.