BB's reaction to Rayleigh's appearence tell us he is really a coward , he is not bold enough and unworthy of being called an emperor of the sea even if he powers are hax and justify his title. he doesn't have the courage and always used underhanded tactics and only fight people only when he is certain that they are weaker than him.
i mean imagine WB,Kaido , BM, Shanks , and Mihawk. would they react that way certainly not.
Luffy might get a surprised reaction but he will not show any cowardice.