Gaban hinted at Luffy not being able to see him before he departs
Evil Shanks left, due to something happening in the Holy Land
Sommers and Killingham don't want to fight Luffy
Loki isn't actually evil nor an enemy
The Holy Knights won't fight Luffy and they won't stay on Elbaf. Whether they succeed in kidnapping the giant children doesn't matter. They will leave for the Holy Land either way.
There was no reason for Gaban to assume Elbaf would become unsafe. Oda was speaking through him, to prime us. Strawhats will not see him before they depart. They will thus not attain the final Rode Poneglyph. So they won't go to Lodestar or Laughtale next.
It is too early to destroy Pangea Castle. Teach is colliding with Shanks, not Luffy next. So the Strawhats will soon set sail to confront the Fleetadmiral.

Evil Shanks left, due to something happening in the Holy Land
Sommers and Killingham don't want to fight Luffy

Loki isn't actually evil nor an enemy
The Holy Knights won't fight Luffy and they won't stay on Elbaf. Whether they succeed in kidnapping the giant children doesn't matter. They will leave for the Holy Land either way.
There was no reason for Gaban to assume Elbaf would become unsafe. Oda was speaking through him, to prime us. Strawhats will not see him before they depart. They will thus not attain the final Rode Poneglyph. So they won't go to Lodestar or Laughtale next.
It is too early to destroy Pangea Castle. Teach is colliding with Shanks, not Luffy next. So the Strawhats will soon set sail to confront the Fleetadmiral.