That basically makes 99,9% of the OP world swordsmen and the WSS dream the most significant path in the story, since it implies that the greatest swordsman has to surpass all the strongest and most important characters in the series.
And indeed, it's justified to ask why Mihawk never challenged Big Mom or other solid top tiers. Vista I can understand. Vista at his peak is around YC3 and at the time Mihawk dueled with Shanks, Vista probably was a small fry on vet level. But after Shanks lost an arm, he could've gone for Big Mom or could've challenged Kizaru. Oda could've made a case and let Kizaru chase Mihawk to prove, Kizaru is not enough as he is a mere swordsman who goes against the WSS.