It means nothing, no one said people can't use other skill sets, and beat a superior swordsman with other skills. Pica literally used sword, then drop his sword and fight with Haki at full power.
Swordsman = skill. No need to add another importance to this shit. Shanks's full power is unknown, sword title shouldn't be a factor over him.