I think most people here severely underestimated the Scabbards.
However, many people are also idiots, overreacting to the tiniest bit of feats and recency bias.
For instance, the scabbards hurting Kaido when Luffy couldn't.
That doesn't mean Luffy is weaker, but rather swords have better cutting power than punches. Plus, it seems like most of the samurai specialize in armor haki. It's hinted they all use adv haki with their swords.