It's because of his Eyes,
Loki have Access to "Infinite" Future Sight imo, meaning unlike FS Users who can see One Future, Loki can see All Possible Futures,
Meaning He knows Exactly what's gonna happen
So imagine He grew up warning Elbaph about Upcoming Disasters & They end up True,
However, his Story is Opposite of Usopp, meaning They both eventually became disliked by their People but for Opposite Reasons,
While Syrup Village Hated Usopp's Lies, Elbaph Hated Loki's Truth
Why? Because i believe somehow Elbaph Warriors were led to believe that Loki's Eyes don't actually see the Future, They actually bring it!
They believe his Eyes are like a Curse or Jinx or bring Disasters
For example, Loki could have Prophesized Death of his Father, not actually Killed him, but Elbaph People believe He Fated his Death using his Eyes. It's like imagine if People knew that Usopp's Lies become Reality, They would blame him for anything Evil He says
This is why They believe Loki can End Everything if He is released, He is simply "The Accursed Prince".
He was born with those Eyes, it's not related to his DF
Loki really was born with "Cursed Eyes", and his Mother obsessed with Omens/Prophecies...
Combined with Harley Text, and don't forget that his Mother is of "Pure" Elbaph Descent, clearly indicates that Loki is Descendant of whoever wrote that Book,
Just like Shirahoshi is Descendant of First Mermaid who is Destined to Lead Sea Kings once more,
Just like Vegapunk was born Smart, which led him to his Destiny to rebuild a Pluton-Like Weapon,
Just like Blackbeard was born unable to Sleep....
... etc
Loki is Destined to Re-Write History once more, (He still doesn't control his Power apparently)
And this is what Giants call "Judgment of Elbaph", as Dorry mentioned how his Fated Loss against Brogy was decided by such Judgment, even though No Natural Phenomenon happened nor were They in Elbaph nor anyone related to them appeared ... etc showing that it's "Fate" related Powers,
Regarding this Harald Story, it seems to me that He actually planned to turn against both WG & Elbaph, i think those Soldiers turned against him because He wanted to consume the DF, something that's Forbidden to even their Kings, and He eventually Succeeded & fled Elbaph, so Loki just accepted to take the blame, and his Journey on the Sea was actually to catch his Father, but apparently Shanks stopped him, just like He would have wanted to stop Ace from chasing Blackbeard,
I mean the guy is still 51 Years Old in Human Years if He is still Alive.....
It's very sus that Oda didn't make him an Old Man if He intends to Kill him & call it a Flashback,
In other words, i still don't buy Harald is Dead,