That's more or less is what I am saying. Kuzan questioned Garp if he'd hesitate, and he himself hesitated with his attack, to which, Garp responded without holding back while calling out Kuzan for not doing it.
Garp is essentially saying you're weak coz 'you' hesitate.