In Oda's mind, Roger, Rayleigh, Shanks and Law are all swordsmen. Like it or not, stop being in delusional denial: that's reality, you have to accept it.
Otherwise it's you against the author.
This comes from the Usopp Pirate Gallery, where fans send their drawings and Oda gives them a comment based on the content. Well, guess what? Oda's comment on this was
"A gathering of swordsmen! Get cut and it'll hurt!". No one even asked him about swordsmen. He willingly freely decided to label those characters like that.
So, all the characters in the picture are swordsmen, by Oda's word. Roger, Rayleigh, Cavendish, Law, Oden, Zoro, Shanks, Mihawk, Shiryu, Hatchan, Kaku, Vista; all of them are swordsmen.
It's not my opinion or someone's theory. It's Oda's word.
OPS... I can hear the rustled jimmies.