That's where the Hakiman theory is getting stronger and stronger evidence. I see Rayleigh as the older gen Shanks. Oda even made parallels between them when comparing their CoC. Now Rayleigh is already seen as a legend with incredibly strong Haki. And his swordskill is also at the top. And Roger, he must be beyond that.
We have Whitebeard, who also uses a weapon. A weapon you would think is superior to Roger's saber (a humongous supreme grade blade). With Whitebeard having extremely good Haki as well (equally clashing with Shanks, splitting his the sky and having his CoC compared to Shanks in Marineford), and with his CoO being also at the top level as pointed out by
@Bogard. On top, Whitebeard gets the strongest paramecia, that is able to destroy everything and shake continents.
And Roger, while on his death bed, terminally ill and holding himself alive with medication as pointed out by Rayleigh, just ties with Whitebeard? If it's no DF, he must have a special Haki related ability that has nothing to do with swordsmanship. Maybe CoC can really do crazy stuff.
And if Roger is classified as a swordsman, him not having a black blade is pretty odd.