On the definition of swordsman ...one piece standards. One is a swordsman if;
1. Stated by Oda to be such
2. Fights primarily with a sword
3. Fights at their strongest with a sword.
These can be individual or a combination.
Under 1, we have Fugi , Law, Zoro, etc
Under 2, we have Zoro, Mihawk, Fugi, Shiki, Roger (could fit here based on facts we know) etc
Under 3, the above mentioned, Roger, Kizaru (from what we know vs Ray), Ray , Brook etc.
Just holding a sword doesn't make you a swordsman. Being trained in swordsmanship at just a particular point in life doesn't make you a swordsman. Marines are automatically thought swordsmanship when they are rookies. Just like Copy and Helmeppo. At that moment, they would be considered as swordsmen since that is their default position as Marines. However, as they grow, they move towards what fighting style they like. Coby became a fighter by focusing more on brawling than swords and Helmeppo decided to fight with knives.
To be one, swords are to be constantly used by the character especially when they are fighting seriously.