99% of people in the verse are using a weapon, because a weapon is an extension and makes you stronger. Kaido uses a club, Big Mom a blade, Whitebeard a bisento (which is technically a blade). For pirate, it happens that blades are the weapon of choice most of the times. But those guys don't dedicate themselves to swords like true swordsmen. Roger might as well use a club if he feels like it. After all he even used a pistol. Or Ray using two sabers instead of his Katana. Imagine Mihawk randomly switching between his Yoru and a saber+pistol like Roger.
If Roger at least had a black blade, we could've argued. But cmon, neither Roger nor WB did turn their blades black permanently.