Because if Imu could actually Domi Reversi someone of Xebec's caliber, the manga would be over ten times.
He could Domi reverse literally everyone right now, mugi included, and kill Luffy on the spot.
So, he clearly cannot. Should not.
Also, plot-wise, there Is heavy indication at least Garp doesn't know Imu and so he shouldn't have seen any demonic Powers related to him outside the general regen every HK/Gorosei has.
Not to mention It would be crazy disrepsect from Oda to Xebec if Xebec was made a "slave" by Imu, his sword enemy, more so when Imu and his goons are after his son and wife's lifes.