I think it is just a trickle down affect of Roger and Rayleighs style, we know those two for sure did not eat a devil fruit and have top tier haki (Obs, Arm, Conq). Old Rayleigh and Shanks were implied to have equal CoC haki (the FI related SBS question), not saying that Old Rayleigh and Prime Shanks are equal.
Shanks and Buggy were just kids at the time and were possibly raised from infants to teens on the Orojackson. Roger and Rayliegh were probably father/uncle like figures for those two and they have seen them compete against the very best (WBPs at least) and Shiki and all those top tiers who had devil fruit powers. And they saw that no matter how OP a devil fruit power may seem the Roger Pirates were the ones standing above all. So when they have clearly witnessed the supremacy of haki of over devil fruit why would they give up the ability to swim and that too as a pirate to gain an ability which is potentially weaker compared to pure haki. We saw in the flashback how Buggy blamed Shanks for accidentally consuming the devil fruit and how he hated that, how he hated not being able to swim (of course Buggy wanted to be able to swim to find a treasure).
So Shanks had first hand knowledge on this subject and I think one of the reasons he sought Mihawk was that Mihawk was also one of those who had not eaten a devil fruit and Shanks believed that he could reach his maximum potential by competing against another pure haki fighter, someone who is more closer to his age and he could probably not go to Rayleigh for training etc. after the Roger pirates were disbanded. Maybe even Shanks wanted to recruit Mihawk like he sought other people of fame (Yasopp etc.) and these people were known to not have devil fruit powers. Thinking of this it might be possible that Shanks sought Mihawk like Luffy sought Zoro as their first crew members and while Zoro had no option other than to join, Mihawk was not in such a predicament and challenged Shanks to beat him in a swordfight and if he was successful then Mihawk would join the crew.
There is one point though which goes completely against all this and that is that Shanks sought people of fame to recruit as crew members and there is no criteria that they must not be a devil fruit user the clearest example of which is Shanks asking Marco to join his crew.
