I meant more the people who were responsible of his 7 defeats. I do think those people can hurt him, but they probably didn't fight / defeated him before.
Kaido is the type of character who acknowledge only people who he views as stronger. He questioned who does Luffy think he is, by saying only few people can fight him, before viewing Roger, Whitebeard, Rocks, Oden in Luffy's shadow
And imo, it's to draw a connection with the few people who defeated him before(he won against Oden due to hostage thing, but considers it as a defeat)