Well it's pretty common and most importantly legit to make a distinction between swordsman and sword user.
Some characters could use a sword once in their life but that's hard to believe that they are swordsmen.
Swordsman = character prominently using a sword, independently from DFs, haki or other techniques.
This makes no sense considering the fact that:
Brooke is a swordsman who has integrated his df powers and music with his swordsmanship. Originally Brooke only integrated his music with his swordsmanship. After getting his df he integrated it into his swordsmanship alongside his music.
You have Ohm who is a swordsman that has integrated sails into his swordsmanship.
Kaku a swordsman who has integrated rokushiki and df into swordsmanship. Originally he only Intergrated his rokushiki into his swordsmanship. After getting his df he integrated it into his swordsmanship alongside his rokushiki
Cabaji is a swordsman who has intergrated acrobatics, unicycling and other tricks into his swordsmanship
Above are just some of the people who integrated other abilities into their swordsmanship and are still swordsmen. So your statement that a swordsman is a person that uses swords independent of other abilities is negated by the evidence provided in the manga.
I can understand saying character who know how to use swords but stopped using swords and started using other fighting styles are no longer swordsmen, however, as soon as they pick up a sword again they are swordsmen. A swordsman by definition is someone that uses a sword to fight. Case in point Sanji is currently not a swordsman. He was trained on the way of the sword whilst a child but stopped fighting with them. However, if he does decide to start fighting with a sword, he will be a swordsman.