So roger whose been travelling for decades at that point is carrying a sword for shits and giggles ?
He uses it. But he does not rely on it.
The same way big Mom uses Napoleon, BUT she also uses her fist, she also uses Zeus attacks alone, she also uses Prometheous attacks alone. She does not rely on Napoleon as her major fighting style.
Roger is the same.
An example is Whitebeard. He uses his Bisanto weapon, but he relies more on his fists than his weapon. I hope it's clear. Roger uses it on some occasions during the fight, but his major fighting style is FISTS. Same as Katakuri, he uses Mole spear, BUT he uses fists a whole lot more.
However, someone like Fujitora for example, he is a DF user, however, most of his techniques, if not all, are being activated when he uses his sword. Even something as rediculous as bring a meteor down (which logically has nothing to do with his sword) HE STILL conveys the move through his sword. Thus making him a legitimate swordsman, and simply by default falls directly under Mihawk (NO QUESTION ABOUT IT). EVEN Oda called him "Blind swordsman"
It's like Oda telling you "he is a swordsman". but you guys say "NO he is not, you do not know your own Manga, let me write instead of you" hahaha
Point is, Roger's fighting style was not shown yet. If you can confirm it to me that he is like Fujitora or Zoro or even Oden who basically use swords for the entirety of the fight, then sure. But if he is just gonna use it for the first clash to give us some hype and split the sky, then he leaves it behind and goes with fists? or if he uses it only for one or two moves, or if he uses it only to give him more options against some special DF abilities who swords can be more affective than fists (like Rubber). Then No, he uses swords for specific situation and it's not his major fighting style, making him not someone to fall as swordsman
you can say that he has good or bad swordsmanship skills, but you cannot give him the title as "swordsman" like Mihawk, Zoro, Oden, Fujitora...etc