Do you think Shiryu is a swordsman?
As for Roger, he literally named his son after his sword, and he wields a Supreme Grade blade. I can't see what else he'd have to show in order to be considered a swordsman. Honestly, after this reveal that he's not a DF user, there's a better argument for Shanks being a swordsman than A LOT of people who automatically consider to be swordsmen.
I'll be honest with you: I don't give two shits about whether a swordsman with a devil fruit is a swordsman or not. It isn't about that, that sounds just so... childish, superfluous to me when the truly interesting stuff here is whether Oda can make something deeper, more nuanced with swordsmanship or not.
For me, wanting swordsmanship to be something more than wielding a sword to fight isn't a matter of power scaling as you may be focusing it. When I say that swordsmanship shouldn't be just about using a sword I'm actually remembering the foundations for something way, way more interesting than that (Zoro and Mihawk's values as fighters, swords being sentient to an extent, Koushiro saying that a sword that can't choose what to cut isn't even a real sword, Zoro coming to the conclusion that he wasn't understanding swordsmanship properly everytime he attacked Bonez like an animal...) and expecting Oda to do justice to his own seeds.
Roger can be whatever Oda wants. For all we know, the author could come and say he had the same power Dragon currently holds, that he was an incredible brawler too, that he was an amazing sniper or whatever; it's irrelevant because I highly doubt he will ever be fleshed out as a fighter, so who gives a damn. The only thing we know thanks to Chinjao's words is that he was on top of the conqueror's as the Pirate King, and that alone isn't swordsmanship, that's having the most powerful conqueror's in the world no matter how he channeled it.