Queen is not a pure swordsman. But he's still a swordsman.
Just like how King is clearly not a pure swordsman, but still a swordsman.
There is no point of having a weapon on you if you're not proficient in it.
So saying Zoro vs Queen is not going to happen because Queen is not a pure swordsman or whatever is just assinine to me.
On another point, Sanji vs King and Zoro vs Queen, or visversa, is the same for me, but there's no denying that the first option is the most likely to happen for now.