He fought Imu a few secs ago, got slammed to the ground, became larger, grew demonic wings and went rampage. How did they not understand there is something wrong with him? I know this is just PIS but Oda really could have choosen a different way to make them leave.
At the very least Oda could have done more of the dialogue thing, like make Imu twist Rocks into pretending that he tricked his crew into joining him for a quest to gain Imu's power.
Maybe even give Imu telepathy and make him manipulate the Rocks pirates emotions to increase their distrustfulness.
So then WB's "were we impediments to you?" and departure would make sense.
It would also cement Imu as a parallel to Luffy, where Luffy's strength is gaining allies, Imu's is sowing discontent.
Really hope something like that is revealed to be at play here.
Because if it's not then, some of the Harry Potter fanfics I've read where Ron is a traitor have more convincing "betrayals" than this one.