The thing is that none of the 9 Scabbards are strong according to Kaido. We've already seen Kiku and Kinemon nearly accept death even though their raid failed and the samurai of Wano accepting death by Luffy. I feel like Kaido's point is more proven than debunked in the story so far.