Man imagine Akainu, Aokiji, Greenbull, and Fujitora all pulling up on the island while Kizaru is still here. Fuji alone was lifting an entire island above his head on Dressrosa, lord knows that would be the biggest wtf moment in the history of this manga for that much firepower to be on one island at the same time.
Nobody even sweatin the gorosei all pulling up, that tells you how people truly feel deep down.
Akainu pulling up alone would garner more fear like you said, and people want to tell me they actually believe in the Gorosei.
Side note this also goes to show you how much Kizaru is actually being held back cause he's a walking buster call/nuke by himself. He has not even shown the casual destruction of past. The excuse we could give him was govt territory, but not post Buster Call, and fair he got grabbed recently and tossed away. But if he doesn't get up and start blowing shit apart soon, Oda will have to do much better at presenting the why.