Not just you.
I think it's Luffy's massive popularity in Japan that helps him won the popularity poll every time, in general. I'm not Japanese, so I don't know how or why his character (or lack thereof) resonates with the Japanese audience.
Zoro is much popular overseas.
That is to say, I think the Japanese audience is likely more hyped for Luffy vs Kaidou, whereas the international audience may well be more hyped for Zoro vs King.