Loki and Harald continued to fight long after the mark had been damaged lol:
Harald didn’t regain control until after Loki lifted his body up after beating the shit out of him lol. So damaging the mark had nothing to do with it.
Loki is simply that much stronger than Gaban and Saint Shanks. They were helpless to free Harald from Imu’s control like Loki did.