Don't nitpick if u wanted this argument
Garp literally said this to Luffy, and what Garp did?
U also missed that it's not him that not holding back, he talking about "this power" which is Yoru. Literally not him. Because the next panel he was talking about how fate gonna play in this, either Luffy escape Yoru or not.
He talking about Fate like Loki.