To you world's strongest swordsman doesn't mean world's strongest swordsman
To me, world's strongest swordsman (or most skilled swordsman) = the strongest swordsman, meaning that their swordsmanship is the strongest / greatest / most skilled, not that they necessarily defeat all swordsmen. For me, the title applies to what Mihawk can do purely with his swords.
Kaido is the world's strongest creature but he was cut by Oden. He was also defeated by Luffy. Oden and Luffy are not stronger creatures than Kaido as far as I know.
Fights are more complex than you are implying. Like I said previously, maybe Shanks can paralyze Mihawk (at least affect him with his haki) and defeat him, while remaining a weaker swordsman.