Anytime there is a major group, and they aren't Luffy's final opponent, Swordsman is always strongest.
Borogue Works
Enel's Priests
CP9
Calamities
If Gorosei are EoS Battles, and Luffy faces Imu, you think Zoro's opponent won't be his strongest subordinate? When has that ever hapepned?