I would just say that it's best to avoid slandering Luffy lol, it wouldn't age well.
Cause this is Oda, who has gone overdrive with pushing Luffy since past few arcs. We know Luffy is the golden boi messiah, it's just a matter of when he will get wanked again.
And even here, we don't know how much power Luffy used either. At least wait until full summary lol. Cause if dude was in base or G2 or something, the slander will immediately fall apart.
Future sight isn't something that is always active. One has to actively use it.
Plus, Gaban was friendly to Luffy till now, it always gets him some time to get going in such scenarios.