They may just be reputations in the story but they're also titles Oda specifically assigned to these characters with a purpose. He gave WSC title to Kaido and made sure that he lives up to his hype. He called WB the WSM but still withholds his hype to this day. Why would Mihawk be any different? Shanks isn't just a random guy who happened to carry a sword around. He was raised by two high-class swordsmen i.e., Roger and Rayleigh, and he's also the biggest rival of Mihawk. Mihawk has little interest in non-swordsmen; why would he bother having a rivalry with Shanks if he isn't a swordsman? It makes no sense.
I can see why people feel Shanks shouldn't be weaker than Mihawk, but most reasons they use to dismiss Mihawk's claim as the WSS are invalid just like the Hakiman argument. They just use whatever dumb loophole they could come up with to say Mihawk isn't as strong as he is because they hate the guy.
Mihawk wouldn't be recognized as the WSM if there are other swordsmen who surpass his infamy. Shanks, in particular, isn't living under a rock. He's one of the most famous swordsmen in the world. And everyone their moms knows about Mihawk's rivalry with Shanks—It's literally a legend in OP world as noted by WB of all things. So if people call Mihawk the WSM, it could only mean one of two things i.e., either the world accepts that Mihawk is stronger than Shanks or Shanks is somehow not a swordsman. There is ample evidence suggesting Shanks is a swordsman while there are little to no implications of him not being one.