Sakazuki himself told GB to watch himself
He knows that Luffy can easily solo his ass.
The Gorosei were scared of angering Kaido despite apparently having 4 Yonko level fighters
Shanks talks to the Gorosei like an equal while Lakazuki can't even sit down when he comes to report like an errand boy