BB did not go to the island at the time of a marine invitation without knowing the risks. He knows what the Navy can do and he knows what Hancock can do. He was still willing to risk fighting them both to get his hands on her fruit. You haven't been paying attention if you think BB would've just backed off as soon as Hancock turns his underlings back to normal. He knew the risks and was ready to sacrifice them if necessary. Rayleigh's involvement is what made things too risky for him, which is the major reason why he backed off.
"he knows what Hancock can do..." He knew to a certain extent on how the Df worked, which is why he used the oppertunity where she was fighting against the Marines to grab her, instead of face to face. BUT he wasn't fully aware of how the DF worked, which is why Oda had Boa explain it to him:
Teach tells Hancock that he has been patiently waiting to get her powers. Hancock tells him that her powers are build upon her own beauty. If Teach kills her, those who have turned into stone will remain in that state. Even the next user of the Mero Mero no Mi will not be able to reverse it. Koby says that Hancock is a problem bigger than he expected.
Teach came in believing that even if some got turned into stone, he'd be able to reverse it, if he just got the DF. But that's not the case. And that changes the entire dynamic. Which is why he then says:
Teach: "
This situation looks like it's not going anywhere, ..."
Then he even asks Koby on his opinion: "
Do you really think if I let her go... she's going to be nice and turn everyone back to normal?"
Hancock then tells him she'll do it if they agree to leave the Island. But then Teach replies... and this is the crucial part here:
Teach: "
And... that is a lie!! The moment I let you go, you will turn us into stone.
And that will be our end, right? To be honest, I think there's no way a man can't be affected by your powers.
It's a difficult situation... So I think I should just kill you."
Mans admitting that if he let's go of Hancock, she would own em. He says it's a difficult situation, where he cannot at all risk Boa being let go from his grasp, that's how fearful he is of her. And says he would just kill her, which remember now results in him sacrificing two of his 10 titanic captains, and unlike the scrubs like Burgess... these two are the ones he spent all the time hunting pirates to become a warlord to get from ID.
Then Rayleigh comes in, tells Hancock to free everybody from the stones. And also essentially guarantees that she won't turn em stone again if they all quietly leave. Now tell me what reasons Blackbeard have to refuse what Rayleigh did?
He already admit he can't take on Hancock face to face and she'd pawn em all (including Koby and em), which is why he got her when she was fighting somebody else.
And the only way to get back his commanders is if Hancock undoes her thing, which Rayleigh being there certifies Hancock will do that and not turn Teach and them into stone. Hence Rayleigh says "I will be witness"... which goes to Teach telling Boa, "that is a lie!!" he didn't trust Boa's words... with Rayleigh it's a different story.
Now in the off-chance Teach decides to not let go over her and tries to kill Boa still. Teach has to risk Rayleigh being fast enough to free Hancock from him before he can kill her. And once she's free... it's GG...
The Navy here are a non-factor, he already knows their hands are full atm to do anything about him right now.
Only thing Blackbeard was scared of this chapter was the mero mero no mi. Which is the reason he backed off. And Rayleigh of course saving Boa, who was dealing with the Marines & BB pirates.