Kaido himself didn't give a shit about all the others who fought him before Luffy, he only used Shuron Hakke, future sight, the alternate hybrid form and Kaen Daiko against Luffy alone, and he specifically acknowledged Luffy as someone who could fight him and Joyboy as the man who could beat him.
Besides, Luffy did like 90% of the damage anyway.