Luffy says I'm going up, not we, it's a hassle to act as a mobility support for someone when speed is a priority lol.
Plus without Zoro they will be outmatched trying to fight all these guys and hold the line.
Could go either way of course, but I'm thinking Zoro is staying on the ground.