I know, but this isn't the first time that they're afraid of Shanks, he has something special.
In Marineford, the Marines tried to claim Ace and WB body's, the Shanks arrived and Sengoku said "I will let you take it, just because it's YOU"
Then 500 caps after, he just entered the Gorosei room.
Either he is strong enough to surpass anyone in the world right now (BB is an exception), or he has some special characteristic, like being son of Xebec or idk... Pure hype...