Personally, I believe those five are cut different in terms of their spirit. Like they are the only characters bar Luffy who we could actually see having a chance at becoming the new Joy Boy.
Garp, Linlin, Mihawk, Sakazuki, Sengoku... It doesn't matter how strong they are, they all feel way less willed, less adventurous, less whatever you want to call it, but they don't feel like game changers. Roger was the Pirate King, Whitebeard clearly stood above the rest but the Pirate King himself and started the current era, Rox gathered the strongest crew and wanted to conquer the world, Oden was a wild beast who dreamed with freedom and died smiling and Shanks has been addressed as the freest man on earth plus is Luffy's role model.
Like they all hit this feeling of transcendence that no other top tier has. We could include Blackbeard here (big maybe, knowing Whitebeard's words), but he's too much of a noob for Kaidou to care. If I had to choose five candidates among the known top tiers to be Joy Boy (or at least someone with such spirit), I'd certainly take the same as Kaidou did. They are dreamers, they seek freedom, they are ambitious. The rest of them are either silver medalist, accommodated or simply Government dogs.
Considering that Kaidou inferred that Joy Boy would be the one to defeat him, we have some basis to look at the "people who could defeat me" statement from a different light than just simplistic power levels.