I think with Gaban tbf to him, unlike us and the SHs he's not really seen Luffy's journey and the insane amount of plot armor this mofo has lol and is just going off what he knows, which is, Shanks gassing up this kid potentially as the man they've been waiting for that has also awakened as the legendary Nika, the so called "savior" of the world and the only one that can actually challenge the status quo being Imu specifically. Also being a Roger pirate and someone that has conquered the grandline, he knows the true history of the world and also knows what is required to actually challenge the world. There's also probably a part of him that also wants to see the continuation of his Captain's dream. It's not like Gaban doesn't have ties to Elbaph, I mean he has a wife and kid on Elbaph and is bascially a citizen of Elbaph himself and even then he's thinking of the bigger picture. Right now he sees this so called "messiah" of the world that hasn't quite mature yet being actively gunned down and targeted by the King of the world, that only this so called messiah can defeat. So he's being logically here and saying run and fight another day even if I and all the others that have been waiting for you for so long are going to die here.
And Usopp is saying "we ain't gonna run regardless" which is fine.
But the whole fck your expectations and destiny spiel as well as the we won't allow you steal his freedom nonsense was forced and unnecessary. It's just all so dissapointing coz Oda's bascially insisting here that Luffy will continue being oblivious to his destiny and that he's just gonna keep scamming his way to the top thanks to said destiny previledges by the way.
There's nothing wrong with understanding who joyboy was, what it's really gonna take to be PK and also the kind of obstacles that are in the way of your so called dream of being the freest man on the sea. When you understand that you can now say, aight bet! shit is peak and I understand what came before me but gonna do things my way. That is called bloody character development but no Oda's hell bent on keeping Luffy oblivous to end, insisting that he isn't going to be swayed by destiny when destiny is quite literally doing so much heavy lifting for him