Oddly, none of the gorosei have shown any fear of Luffy, who beat Kaido.
Saturn gets pancaked, flies back to continue throwing hands. Ju Peter tries to eat him. Warcury straight-up tries to prevent the gang from leaving, before he attempts to attack Luffy and asks why the giants are protecting him. And the panel posted literally proves they feared Nika being awakened more than they did Kaido.