I don't think it's about who can hurt Kaido.
Do we then argue that the Admirals or Mihawk could not even hurt Kaido ?
Or Big Mom who fought him equally and is perceived as his peer would be unable to hurt him ?
I think it's more about something in Luffy reminding him of those people, since it's a follow up to his own words towards Luffy :
" Who do you think you are"
And the answer to that can be found in the resemblence Kaido sees between him and those shadows of the past.
I meant more the people who were responsible of his 7 defeats. I do think those people can hurt him, but they probably didn't fight / defeated him before.
Kaido is the type of character who acknowledge only people who he views as stronger. He questioned who does Luffy think he is, by saying only few people can fight him, before viewing Roger, Whitebeard, Rocks, Oden in Luffy's shadow
And imo, it's to draw a connection with the few people who defeated him before(he won against Oden due to hostage thing, but considers it as a defeat)