World Government has no reason to know the Straw Hats are on Elbaph, and they're already resorting to kidnap children. I'm sure their plans would be enough for the giants (and that's already hype given it's the strongest island), but I don't think they've incorporated Gaban and the Straw Hats in their plans. Wouldn't be surprised if things escalate once the Holy Knights realize the crew is here. I doubt three Holy Knights are beating the whole Straw Hats.