If Mihawk wasn't the WSS and tied to Zoro, then these deniers would readily accept Roger as the previous WSS.
I get it that it doesn't fit your agendas and headcanons that Roger, Shanks, etc are swordsmen, but they are.
Roger is the strongest swordsman we know of from the previous era.
Mihawk is factually the strongest swordsmen of the current era.
Zoro's ceiling is very high.