Kaido wanted more than anything to get the chance to finally beat WB before he died, yet Shanks sent him running back to Wano empty handed empty handed while Shanks was the one who got to achieve all of his goals at MF.
We then saw in Wano that Kaido holds Shanks in the same light as Roger, WB, Xebec, and Oden, the 4 people who clearly handed him his L’s in the past. We literally saw Oden give Kaido an L on screen, Kaido’s dialogue basically confirms he lost to WB at least once in the past, and you’re straight up delusional if you think Kaido never challenged Roger or Xebec.
If Kaido was only including Shanks because he’s heard tales about Shanks’s strength, why were people like Garp, Sengoku, Rayleigh, BM, Shiki, etc not also included? Given that the other 4 people in his “top 5” have pretty clearly beaten him in the past, it’s kinda obvious that Shanks was included because Kaido thinks he suffered a defeat at the hand of Shanks. They may have not had at straight up fight at MF, but maybe they flexed their CoC against each other and Shanks’s superior CoC made Kaido decide to back down, which Kaido considered a defeat.