It's not the timing relative to the arc, but relative to everything else that needs to happen for Oda's story to work.
Luffy meets Loki, leaves, then comes back just in time for the HK to miraculously decide to make their move now instead of any other time in the past 6 years.
The plot doesn't start unless the SHs are on the island.
It's like the whole world is made out of toys that only come into play in proximity of Luffy, while for the rest of the time stay dormant.