Given the reveal that Imu is one of the 20 kings, my guess is that he’s from the Figarland family.
This is why Shanks is allowed to speak to the gorosei directly.
Due to Shanks being from Imu’s clan, the gorosei gave him special treatment whereas other ex-CDs like Homing were cast out.