We can agree that Xebec and later Roger was much stronger than current Gaban but they knew that there is no way that could do anything to Imu. And even this arc he is there for a single reason there to push Luffy toward more power Luffy and Loki were mean to know despair and helplessness so their Haki can bloom and they will be able to know their devil fruits better many Loki will awaken his DF. And Imu will just have to retreat because of the plot.
Oda already prepared many reasons.
Imu time limit
Backlash of losing convants
Luffy and Loki will power up
Decent fodder(from Imu's viewpoint) like Gaban who can delay a little