People need to realize that this is no different than when the Straw Hats teamed up against Oars. The gap between him and them individually was miles apart, but since they were working together they were able to hold their ground against him for a bit before Nightmare Luffy had to come in and save them.
I think the main issue here is that Oda is using the scabbards to damage Kaido before Luffy and him fight, rather than the people they predicted that would do it. We all know damn well everyone was onboard with a team up to weaken Kaido, and now that it's happening, it's all of a sudden an issue.