Depends on how Oda defines that whole Swordman thing.
I would say, if someone uses a Sword as one of their main Fighting styles and is very skilled with it, you could Count them as a swordsman even if they are a hybrid. You stil can have over ability's like a DF for a example and fight with fists, legs, etc.
Fujitora is said to be a blind swordsman (if i am not mistaken), does it mean Mihawk is stronger than him? Not necessarily. Mihawk might be better with the Sword as Fujitora, but Fujitora also has a DF soooo...