A lot was already said, but I'd add that you have to keep in mind that a guy like Ray, a top tier swordsman, is able to kill these things, but their sheer size makes it nearly impossbile for non top tier brawlers to kill them and in general it's just really difficult to stop them.
You already have Rusukaina beasts, that are like 1/100 their size, who could hang with Boundman Luffy mid timeskip.
Some are multiple kilometers tall, long or big, so if you don't have the tools necessary to inflict significant damage onto them or stop them, there's nothing one can do and that affects even the majority of high tiers.
High tier to top tier swordsmen, top tiers in general and otherwise only guys with extremely high fire power such as King and Queen are nearly the only ones, who are even able to successfully do something against them/one of them.
Good luck with an army of them and it's also like some before me said: they can easily create floods/tsunamis or sink islands. There's barely anyone in this verse that could stop them from sinking an island.