A person holding a sword is not always swordsman btw. Just someone who is skilled with it.
For example, Luffy is not a swordsman.
	
	
		
		
	
	
	
	
		
		
	
	
	
	
		
		
	
	
Kaido is not a swordsman or a gunner.
	
	
		
		
	
	
	
	
		
		
	
	
		 
 This is a pretty bad comparison. 
Luffy isn't designed with a sword. He isn't a swordsman. 
Luffytaro is pretending to be a samurai, hence why he has a sword. 
Is Luffytaro (meant to be) a swordsman? Yes. 
But we all know Luffytaro is simply just Luffy, who isn't actually a swordsman. 
Kaido isn't designed with a sword, he's designed with a Kanabo. 
Luffy dressed up as a swordsman and Kaido used a sword in a very particular instance.
These aren't good examples to bring up for a "swordsman" debate.  
Roger, Shanks, Law, and Fujitora are all designed with a sword and all fight with a sword. 
They're swordsmen. 
Another common "counter-example" I see is Rayleigh. 
Some consider Rayleigh to not be a swordsman because he's used haki with his hands. 
Is Hygoro no longer a swordsman then? I would imagine very few people think of him as a "hakiman".