Yeah no one was forcing anyone to think of a 1v1. Roger, Rayleigh, Gaban, Oden, Whitebeard all had the opportunity to work together. Once they figured out that Imu wants to sink the world and kill everyone, they easily could have recruited Garp, Harald, probably Big Mom, Shiki maybe even Kaido and Sengoku.
All that to avoid fighting Imu. Though it makes sense, given that Imu low diffed Gaban in one chapter, and might kill him in the next