The Red Hair Pirates could retaliate, but from everything we’ve seen of them, they’re the least aggressive Yonko crew, or at least don’t want to fight if it can be avoided by some means. Once they learn of Bartolomeo’s association with Luffy, it’s even less likely they’ll destroy his crew, but they may rough them up for show so the people living in their territory feel safer.