To be fair, its heavily implied that the battle has been going on for quite some time already. At least the time it would take to get all the prisoners to board the ship, set sail and be a decent distance away from the coast.
Garp and the boys have seemingly been going at it for that entire time while overwhelmingly outnumbered and going by how the chapter ends it is not quite yet over for Garp.