What are the odds that all 5 Gorosei end up near the giants ship? Nusjuro is already there, Saturn is close to the Sunny which is currently above that, 2 Gorosei are following Luffy there and Mars can fly.
That's one way the iron giant can get all 5 at the same time.