I think Oda's strange love for making the World Nobles be the most evil people in the world backfired on him this time, in regards to Garp.
I think their hunting game should have been something where they just hunt the slaves they already have, and just choose a random, out of the way spot.
That way, the Marines would keep the townsfolk out of reach while it's happening, but then Xebec would decide to kill everyone on the island. Then Garp would at least have the chance to protect the citizens of the island who had nothing to do with the games.