Dorry and Broggy are next to the castle so they seem the most likely.
Either them or Sanji, he was talking about protecting Nami earlier in this chapter and while this is likely just his usual simping talk, maybe this is a foreshadowing of their group encountering NotShanks and Sanji staying behind.