The point is that people are unwilling to dissasociate swords from haki from some reason.
Haki is literally a superpower that can make you hard as steel, make barriers, knockout people, do weird chiwaves where you dont even touch, sense oponents, see the future, kill future observation.
But as soon as you pick up a sword, Haki fuses with the swordmanship completly.
It's impossible in the head of Mihawkstans that someone can have a B+ in Haki and A in swords or vice-versa, to them Haki is swords.