I mean, the last time they went to war with a Yonko, Whitebeard completely wrecked their headquarters right in front of them. Not to mention, he devastated the island they fought so hard to protect, all while dealing with illness, old age, and heart attacks. Attacking a healthy Yonko would be much worse than that.
For instance, Kaido would obliterate Marineford in a single strike, a magma dragon the size of an island with continental-level destructive power and the ability to inadvertently melt massive structures upon contact, accompanied by an army of 30 thousand soldiers including the mighty samurai of Wano, along with warships equipped with the world's finest precision weapons manufactured in Wano that is an exaggeration for the Navy, the statement from Green Bull admitting to fearing Kaido wasn't a coincidence. He alone is making four admirals work together to stop Kaido's final attack. LOL.