Imo Admirals are reserved for his crew (Kizaru here, the remaining trio sometime later), but the Gorosei are reserved for his allies. Zoro, Sanji, Sabo, Kid and Law do seem to equate to Luffy's greatest allies, so I would imagine that these would be the people to take on the Gorosei at that point. Imo like Garp, Dragon will eventually show off, but meet his match in the end and Sabo takes over. I assume Dragon will lose to Imu at some point, then its Luffy's fight in the end.