Yeah, He indeed said He won't Lose to anyone who calls himself a Swordsman, but did He say "It's Fine if i Lose to Others"?
After He lost to Mihawk, what did He Promise?
Also, why take One Statement & ignore the Rest as if Zoro's Dream cannot be Expanded on,
It's like saying Pirate King means finding One Piece, it doesn't mean finding Four Road Poneglyphs & finding Laugh Tale & getting a Crew needed for the Job ... etc
It's obvious that if No One considers you a Swordsman then you aren't in Conversation for WSS, but an Achievement isn't Defined by who Participated in it (For example, who is Sanji competing against to Find All Blue? or other SHs & their Dreams? No One, but it doesn't change the Fact that No One in the World can achieve them, regardless if They tried or not, except SHs)
WSS is Defined by it's Requirements, that's what lets People know if it's a Dream worthy of being among SHs Impossible Dreams or just something that many can achieve,
Do you want me to tell you what Requirements Zoro discovered?
Since you mentioned his First Ever Fight against Cabaji, why don't We look at his Future Fights,
Zoro would have Lost against Hatchan if his Sword was Lacking,
Zoro would have Lost against Daz if his Lethality was Lacking,
Zoro would have Lost against Ohm if his AP/Strength was Lacking,
Zoro would have Lost against Kaku if his Technique was Lacking,
Zoro would have Lost against Ryuma if his Attack Speed was Lacking,
Zoro would have Lost against Hyouzou if his CoO was Lacking,
Zoro would have Lost against Pica if his CoA was Lacking,
Zoro would have Lost against King if his CoC was Lacking,
Do you see now what are the Requirements for Zoro to achieve his Dream & Live true to his Words?
It doesn't matter who calls himself Swordsman or not, what matters is that Zoro cannot achieve his Goal if He doesn't Perfect those,
So anyone who relies on a Sword, are Lethal with it, rely on their Power/Strength & Speed, have Techniques They Mastered, can use any Type of Haki are Automatically Zoro/Mihawk Victims, regardless if They wanna call themselves Swordsman or Wifi-Men,
As for King, Zoro was annoyed cuz King tried to Win without relying on any of those, (Trap Weapon, Trying to Push him Off Island)
Dirty Tactics are not within WSS Path, that's why Zoro said that,
Even Cabaji himself, He was relying on Dirty Tactics (Including getting Help from Buggy), but once none of his Moves work, what did this so-called Swordsman say? He said He is gonna Win using his Swordsmanship, so Cabaji is Separating Dirty Fighting from Real Swordsmanship & even though He relies on it, He still is a Swordsman, which same applies to King, and also Ohm who was relying on his Iron Trial, and also Hatchan who also can Fight without Swords (Literally his Last Technique before Losing was without Swords) ... etc