It is literally stated that Kaidou and King face all the red haired pirates (all of them). (all of them).
Through secondary materials we have information that Kaidou was only with King.
It is explicitly stated that Kaidou and Shanks fought.
2 years after the incident: Kaidou is labeling Shanks as being unable to defeat him, and King declaring himself undefeated.
(With this information you can forget that something serious enough to defeat someone happened during their encounter.)
Shanks is someone who has the ability to fight Kaidou effectively via ACOC, Kaidou is beating him around high to extreme diff, and even after beating him he would have to face all the RHP who were in even greater numbers, it would be something exhausting for Kaidou and King alone, so it would be useless to proceed to MF and get there tired, however much I think that a Kaido even tired can kill that Whitebeard that was shown to us in MF. LOL
The military power that Shanks was in was able to make Pre-Yonkou Blackbeard with the most heinous criminals in the world on his side to retreat and assume they weren't ready yet, and finally make the navy follow his criteria,the fact that Kaidou had the courage to clash against such military power, with only King at his side, demonstrates the nature of the beast.