He is not a swordsman, though. At least not in sense of Zoro and Mihawk.
I don't think you understand what a swordsman is. A swordsman is anyone who is good at using a sword. Kizaru is good at using a sword, hence he is a swordsman. Period. That is the literal dictionary definition of that word.
If I were to say:
"Kizaru is a good swordsman, but he is a much better in other styles of combat."
That sentence would make sense because being a swordsman and knowing other styles of combat isn't mutually exclusive. Law is a confirmed swordsman and we know he is also trained in martial arts and with the gun.
People who act as if you can only be a swordsman if you only use a sword are being intellectually dishonest. They would never say such a thing in the real world and are only acting that way because they think that whether or not someone is a swordsman affects Mihawk's power level.
That is not true. It is possible for a character to be a swordsman and be stronger than mihawk if said character is not as good of a swordsman. Take Big Mom. She clearly is a swordsman but her strongest techniques are weather manipulation hence you can make the argument that she is stronger than mihawk. In a sword duel, she would lose against mihawk but in an all out death match, she would win.
and Oda has never validated this argument that the only swordsmen are pure swordsmen. He has confirmed that both Fujitora and Law are swordsmen.