Because he didn’t fight Imu?
Rayleigh and Gaban didn’t fight Imu either. There’s really no evidence to suggest Kaido was stronger than Ray or Gaban given that he nearly died against Oden.
Gaban is afraid of kaidou, and Oden fell to a base kaidou without even using Acoc.
And he was holding back his strongest forms, while Oden's strongest attack fell short of knocking Kaidou out. Oden has a better portrayal than Rayleigh and Gaban. He's directly compared to Roger and placed on a par with Roger, who, by extension, is stronger than Rayleigh and Gaban. Hell, Oden's sword style was better than Roger and Rayleigh's.
Kaidou is stronger than Shiki, and stronger than Rayleigh and Gaban, after all, Kaidou was being called the embodiment of strength itself and the strongest in the world while these guys were all at their peak, in the 1v1 bet on Kaidou and not on Rayleigh, Gaban and Shiki that's how Kaidou is a bigger monster than them.
Shiki, Rayleigh, Gaban, these 3 guys were never even considered the strongest in the world, Kaidou however became the strongest.