Usopp will find a way to topple them, imo. He and Nami ran off with Tama. Now that the Beast Pirates have identified her, the Numbers, who aren't SMILE users, are the perfect subordinates who are strong enough to handle her converted Gifters and take her out. This will force Nami and Usopp to defend her now.
Plus Usopp has mentioned through a lie how he beat giants, and he'll fulfill that lie like he usually does. Plus he needs something after failing to take out P1.