King is significantly weaker without his sword (his physical attacks are mostly with sword, his strongest fire attacks require sword to channel).
But yeah, the rest you mentioned don't face that issue
With King, I think the issue was that he used swordbreaker technique, which for Zoro disqualifies him as a swordsman.
Honestly I don't understand it, cause King himself is proud of his swordsmanship and enjoys it, he just doesn't restrict himself to it. Some honor thing I guess.
Anyway, IMO safest characters that go into swordsmen are sword users with no DF - Roger, Rayleigh, Shanks, Mihawk, Zoro, Vista, Cavendish etc are all swordsmen. Zoro getting advanced Conqueror's haki, with reasoning that WSS is a dream that requires Conqueror's haki, basically killed any hakiman arguments.
Having a DF doesn't necessarily disqualify too from being a swordsman too, since Kaku and Seraphim Mihawk are swordsmen as well. Fujitora is a weird case, he's called blind swordsman, has "gravity swordsmanship" etc, but I say pulling a meteor from space doesn't come under swordsmanship.