Shanks and Mihawk had their duels, and Mihawk won.
That doesn't mean that Shanks is not an emperor, and wasn't in Roger's Crew.
Maybe Roger was the strongest swordsman before Mihawk became a pirate, but after he died, he inspired Mihawk to become as great as him, which happened?
When it comes to Zolo, he was not a pirate before he was in the straw hat pirates.
Him and Luffy look up to two different people.
You dont haave to be a pirate to become the strongest swordsman.