Then You have not read One Piece chapter 0
When author tells me something about his story I will believe it with no doubt unless I wanna read some other story.
For you to say Roger is not a swordsman you are basically telling Oda to write story differently.
Its his story