i think many people are having a misconception here. BB backed out bcz there is rayleigh joined group , not solely bcz of rayleigh. The opponents side have rayleigh, who can fight BB for some time. In the mean time with boa will defeat most of his captains ( only shiryu ?). On the other side we already have marines with their new SSG. If they helped Boa to take out BB commanders ,then you can expect BB(yonko) crew has already fallen. BB is smart one when he saw rayleigh, he knew that this will not end well for both sides if he pushed them excessively.