The fact that the entire Gorosei is on the island is nothing more and nothing less than the presentation of what Luffy will have to abolish. Luffy has his dream, and we don't know it, but his dream is contrary to the status quo that the world of One Piece has, and when the 5 leaders of the world appear, Luffy, by his instinct, will know that they must disappear. This is the consequence of Nika's awakening, Luffy was unaware of the existence of such beings, perhaps he was even informed of the existence of IM Sama. In short, the Gorosei will retire because something will happen, and they will become Luffy's main focus to achieve his dream. Let's remember that being a pirate king is just the prelude to Luffy's final dream.