Kaido was alone with King, Shanks had his entire crew. We know nothing of what happened other than that.
Also Kaido was hailed as the strongest pirate by Kidd and Killer who not only had an encounter with Shanks prior and lost an arm to him but then made an alliance and targetted him again.
Oh and Kaido thought that no one could defeat him, despite Shanks being alive and them having had an altercation.