Didn't people realize all the people who could hurt or stop Kaido in the alliance at the current time were removed from Oden's castle or far away that they don't confront him?
People who stayed at Oden's castle or confronted to Kaido
Luffy
Law
Sanji
Nami
Brook
Chopper
Shinobu
Momo
Carrot
Penguin
Shachi
Bepo
People conveniently removed situation:
Zoro(He is the Strongest Swordsman in Wano, was off drinking and gambling.)
Kin'emon(Would've diced that Boro Breath to save Momo, but he was running to O-tsuru)
O-Kiku(Would have lingered on Kaido's scales, she was also running to O-tsuru)
Franky(Big Mom was scared of his attack, was somewhere)
Ashura(He left of his own will)
People who stayed at Oden's castle or confronted to Kaido

Luffy
Law
Sanji
Nami
Brook
Chopper
Shinobu
Momo
Carrot
Penguin
Shachi
Bepo
People conveniently removed situation:
Zoro(He is the Strongest Swordsman in Wano, was off drinking and gambling.)
Kin'emon(Would've diced that Boro Breath to save Momo, but he was running to O-tsuru)
O-Kiku(Would have lingered on Kaido's scales, she was also running to O-tsuru)
Franky(Big Mom was scared of his attack, was somewhere)
Ashura(He left of his own will)