Kaido said only samurai. Samurai are the one who learn kendo, bushido code and uses katana, are japanese. Not hard to understand. Zoro, Oden and scabbards are based on japanese samurai.
Roger and Mihawk don't follow bushido code, they don't have a katana, probably didn't learn the kendo.
And no samurai doesn't mean swordsman. Swordsmen from western countries are not at all samurai, they are just swordsman.
Thought it was easy to understand, or are we gonna change the definition of a swordsman which exists since 2000 years for a manga ? Lmaoooo