It doesnt refer to swordsmen only because Zoro doesnt fight only swordsmen.
And he brought back that promise after beating a guy who doesnt consider himself a swordsman.
And why would Zoro think it is ok to lose against non-swordsmen in the first place?
It is just copium among anti-Zolobros...
Zoro ran a gauntlet without heals in Skypiea. Enel is simply where he stopped.
Just like Kaido is running a gauntlet in Wano and Zoro is where he stops.
He wont lose to Zoro just because it was Zoro who finished him off.
Same like Zoro didnt lose to Enel just because Enel finished him off.
People are completely ignoring everything that happened before Enel...
Agenda Piece is as strong as ever.