Charlotte family killed Pedro, tried to kill Sanji, tried to kill Chiffon, tried to kill Vinsmokes, etc.
Even came to Wano and decided to be enemies.
Big mom might even come back in Elbaf and attack the crew again.
Would be ridiculous for them to join with them after everything.
Would honestly be better to team up with Akainu and marines to save Garp, even they are more trustworthy and much stronger allies than Charlotte family fodders.