Would the others follow Luffy if he went to go handle someone he can beat comfortably? What's the purpose in that? Heck no one followed Luffy when he told them he was going to 1v1 Kaidou no, they all listened and fully believed Luffy was gonna whoop Kaidou's ass? The same Yamato that left Luffy for Kaidou to fight 1v1, thought to call everyone over.