Is anybody starting to have questions about Harald's horns and why he " removed them " ? At this point in the flashback every humans he crossed ( after he started acting more wisely ) didn't mention a single thing about them. Oda could have shown that people were scared of ancient giants or smth.
I think Imu or the HK are the ones who asked him to remove them in order to do a pact. They are the only ones who have reasons to be afraid of ancient giants, ( 800+ years ago).
It's possible Imu was afraid of Rocks getting Harald to join him, either as a pirate or as a nation, and that's exactly what Imu is doing with Elbaph rn, because he's afraid they join Luffy/ the D clan.
+ In the discussion with Jarul, Harald understood the government probably had to fight the giants long ago, and that's why they don't want to recognize Elbaph as a country. I guess Harald took it too far and didn't understand the WG higher ups don't care about his ideals until too late.