People don't understand this simple concept: it's a 1 way street. Like
@HA001 says, Swordsmanship is haki. Sure. Cool. Fantastic.
Haki is NOT swordsmanship however lol. You can be a master Swordsman with proficient haki and one of the strongest people in the world. You can also be the strongest haki user in the series who happens to use a sword when he fights, which is quite literally what Kaido is saying here with DFs.
Luffy's fruit heavily uses Haki in all forms, and requires it in G4 and G5, but Haki output has nothing to do with his power overall.
In fact, Luffy beat Katakuri in the opposite way Kaido is talking about here. He had weaker CoA and FS, yet still won due to his proficiency with his DF. At the peak, for someone like Roger, this does not matter.
Realize that Mihawk is never talked about in any of these situations lmao