If someone fights with a sword, they're a swordsman.
If someone fights with their e.g. fists, they're a brawler.
If X fights with both their fists and a sword, they're a sword(wo)man and a brawler.
If X cannot even cut steel, but can throw punches at mountain level, they're still a (very weak)...