AdCoC being the weakness of Imu and the Gorosei makes the Gorosei somewhat special, just like Kaidou. After all, as seen, Joyboy's Haki could affect Imu and cause him to lose control, thus deactivating his powers. This could mean that only characters with AdCoC have the ability to surpass the powers of the Gorosei, and consequently, they are the ones most likely to defeat them, contradicting their supposed immortality. For example, Big Mom has a better chance of defeating Saturn than Kizaru, since Big Mom has AdCoC, a mechanism that works against Saturn.