Official sources states both King and Fuji are swordsman. Fuji has been called a blind swordsman.
King and Kaku both use kicks. One doesn't consider kicking as part of his swordmanship style while the other does. That tells us that a swordsman is not defined or limited to a certain style. This is why the pure swordsman doesn't exist in One Piece. King was talking about his fighting style, not title. That tell us title doesn't matter to him because he is going to use whatever style, power, and ability he can in a fight.
Unfortunately, some have failed to understand that for years. Mihawk is not limited to just swordsmanship. When Zoro first encountered Mihawk and challenged him to a battle and for his title Mihawk pulled out a pocket knife. The WSS defend his title with a knife. Think about that for a minute.