Whitebeard, sure.
Roger, we don't know. Calling him swordsman is about as vague as calling him non-swordsman. He used a sword- and we don't know if he had anything else to his fighting style, though it's confirmed he had no Devil Fruit.
Meanwhile, there's a lot of support to call Shanks a swordsman- him having no other ability, using sword against almost everyone, having duels with WSS, etc, and absolutely nothing to suggest he's not a swordsman.
You're taking characters like King, Fujitora, BM, etc- who use a sword but have much more going for them, and use it to generalize character like Shanks as non-swordsman.
That's just retarded. It's like using the phrase "not all cooks are fighters" to say "Sanji is not a fighter because he's a cook."