I actually had a thought on that. You know how Imu hasn't really......shown any mind control powers? Like, at all? He has telepathy (or the Voice of All Things?), but he hasn't shown the ability to read the minds/memories of his subjects, no memory manipulation or anything (despite people thinking he MIGHT have done that to Gunko/Shuri, he has shown no sign he actually CAN do that) and he also hasn't shown the ability to control anyone's actions outside of the Covenants/Domi Reversi.........and Harald didn't start to change until AFTER he made it home to Elbaf, despite presumably having the Mark for weeks at that point (minimum; however long it took to sail from the Red Line to Elbaf).
I think the Covenant doesn't actually warp your mind unless you try and defy the orders Imu gives you. Harald refused to obey, and the first thing that happened was his body started to act without his own will to fulfil his orders. He continued to fight it and tried to give orders for people to stop him, and then what happened? His body started killing people who were getting in the way of him doing what Imu wanted. The intensity of what his body was doing against his will increased. He KEPT fighting it, and then and only then did his mind start to change. He started to rationalise his orders, to become the kind of person who WOULD obey those orders.
But only after fighting desperately to NOT do something he was told to. So I think that most of the CD's have never once actively disobeyed a direct order, so the Covenant wouldn't change them at all.