Blackbeard was intimidated by Rayleigh after not giving a shit about Boa. Though Blackbeard > Rayleigh according to Rayleigh himself, judging the situation, it would've been a tough fight, hence Blackbeard saw no point in continuing. And Rayleigh himself repeatedly points out that the issue is with him being old. Indeed, a 78 old retired man who spent decades gambling, drinking and fucking. Not only is his body rusty, but also his Haki, since we know Haoshoku = ambition. So a prime Rayleigh with much higher ambition would also have stronger Haoshoku.
And it's not that Rayleigh is a bad match up for Blackbeard, but that Blackbeard is a bad match up for DF users. After all, Teach has a top tier Logia, the Gura fruit and Haki. Even if his Yami Yami no Mi doesn't unfold its full potential against DFless people, Gura + Haki in itself is devastating. I'm sure Blackbeard has Haoshoku too. He could sense Luffy's Haoshoku pre TS already and he is one of the top dogs in the world, while Haoshoku is common among those who have made a name for themselves.