If you don't go to a tea party mama sends her assassins after you
Luffy would have had to face her either way. Not to mention the tamatebako would have eventually set mama off.
If this is anyone's fault, it's Luffy's. He didn't even need to go along with the yonko assassination plot, they could have just got the vin smokes, and got them out of there before the tea party started. Not to mention it was he who challenged Big Mom directly at FI