Luffy was rushing, climbing stairs. Along with Jinbe, and Sanji. Zoro was rushing too to Rooftop too. Only back then they were interrupted. Luffy and Zoro were rushing but then bite by Queen and slammed down. Sanji was distracted by Maria, Jinbe by WSW, but ultimately they were still in hurry to the Rooftop.
Here against GB, same 4 characters are watching from afar, sitting comfortably. They don't rush.
While it maybe true that they could interfere if things went worse, that didn't happened. Because Shanks scared tf outta GB from picking the new leaf, just like he scared tf outta sea king from picking new leaf Luffy.
I thought CoC was made to scare animals and fodders, never thought it was made to scare Admirals.