Sanji's panel this chapter is fishy to me. Like BangMi mentioned, what Sanji said should've come with the typical thirsty face, but it didn't. We got a cool shot of him instead, with a curious expression.
In this chapter we got Nami, Usopp and Chopper looking to Sanji to get them through the trouble, and thus highlighting this trait of Sanji that has come to mean so much to them. They're putting their trust in him, while of course also showing their worry about the prospect of Sanji+brothel. Because they know him too well.
Isn't it odd that Oda is first directing our attention to this savior aspect of Sanji's character, only to then have him just run off to prostitutes?
Two chapters ago Oda had Sanji go all gaga over the prostitutes, so that when we get the line from Sanji this chapter there would be little doubt where he's heading off to. Oda is setting it up like that.
All of this makes me think, what if Sanji's lying about where he's heading off to, as a way for Oda to subvert our expectations? Note that it's Carrot's flashback, so Sanji isn't actually lying to Nami. What if Sanji's planning on something?