I mean, it's stated and shown that he is negatively affected outside of MJ and can't remain beyond it's boundaries for too long, so yeah, it's a factor. Let's also not ignore the fact that Gaban is a more experienced fighter with a higher technical mastery of CoC. It's the same ability he was lecturing Luffy and the others about in the context of fighting these immortals, which includes Imu. The fact is, Imu isn't done. Luffy, and probably Loki too, will fight him again, and he'll probably still have the upper hand. This moment is inconsequential to his established power level. All it does is give respect to an old legend who just lost his arm this chapter and might lose his life in the next. Oda was never going to humiliate him with a piss poor performance.