Other than him simply being cooler and a beloved villain, my assumption is based on Lucci being YC1-ish compared to the others being random VAs (that's their portrayal).
Laugh all you want, but there's no reason to believe any of them is a decent match for Zoro, so if Zoro is not engaging Kizaru, he may as well just fight Lucci better than those guys. I don't need to know if he's a "legitimate matchup" to say that.