Stop over-complicating things
if a character almost exclusively uses a sword then that character is a swordsman. no one else is.
the debate only came up when Mihawk wankers tried to elevate Mihawk above EVERYONE.
if Garp accidentially sat on a sword once they would label him a swordsman aswell. Luffy is now a swordsman if they would have it their way.