On one hand, I want fights as well, on the other hand, Law's submarine hasn't even landed yet, Zoro just got lost and that plot line (whoever Oda wants Zoro to meet) hasn't materialised yet, the Denjiro and Kinemon's forces are still moving into position, if they get caught now, it's really, really bad. I think there's a real sense in which it is too early.
I'd also like Luffy to not be so reckless and maybe exercise restraint when the situation really calls for it (like now).