It is how it works though. The WG is still aiming to get rid of Luffy, but they gave up on Doffy. It wasn´t even like they looked the other way or something, he was able to summon cipher pol like they were servants, Akainu rightfully points out how humiliating it is. The Gorosei couldn´t even make an argument, they just had to use ad hominem and whataboutism. Like it or not, the manga fact is that Doffy made the celestial dragons his bitch.