It's TCB's translation that made it seem like he was fearing Rayleigh due to his
reputation. The viz implied that it was just the situation(
his crewmates turned to stone with no way to turn them back)
And Rayleigh played the role of mediator in it while knowing that if not the situation, he had no chance to win a head on fight at his age, so with no possibility to save Hancock at that point