Blackbeard is brave but he's not stupid. His decision-making comes down to the odds and what serves his purpose.
I don't doubt Blackbeard would have defeated Rayleigh, but what state would that have left him in? What cost would fighting Rayleigh bear on his plans? Predators in the wild avoid each other because the injuries sustained even in victory could be costly or fatal.
BB is a man that stayed at Whitebeard side for decades, waiting patiently for his opportunity. I doubt he would engage Gaban under the same circumstances for the same reason.