I think it could have been a few different things.
One, it was unexpected. Blackbeard is willing to take some risks, but he likes to know about them beforehand and prepare. Supposedly, he didn't expect Rayleigh to show up, so he probably never developed any countermeasures for dealing with him. Blackbeard has nothing to gain from fighting Rayleigh, so it probably wasn't worth the risk to him.
Two, Rayleigh doesn't have a Devil Fruit. One of Blackbeard's greatest powers is being able to nullify Devil Fruit powers, thus forcing the Users to fight without the powers they've come to depend on. Putting them at a disadvantage instantly. Rayleigh doesn't have that. That means Blackbeard doesn't have that advantage in this fight. Which again means, it's probably not worth facing him.
And Three, and probably most importantly...Rayleigh's actually SMART. This is actually a pretty rare attribute in the One Piece World. A lot of the characters in the manga are kind of meat-heads. But, Rayleigh actually analyzes and thinks through his battles like a pro. It's possible that Blackbeard doesn't want to let Rayleigh see how he fights, and find out some sort of secret about Blackbeard or his powers. Blackbeard keeps many secrets, especially about the particulars of his powers. And Rayleigh is one of the few characters who may actually be able to figure out what they are. Rayleigh might lose the fight, but he might be able to figure something out and get word to someone else. Putting Blackbeard at a HUGE disadvantage in the future. So, again, not worth risking it for Blackbeard.