Imo because He was born with Special Eyes,
Ones that can see the Future unlike anything we've seen before, but i don't know how they function, all i believe is that They are FS on Steroids
But i think Giants started to believe that He doesn't see the future, He actually brings it, it's like He is Cursed or a Jinx,
It's like They don't believe his Eyes see the Future, they believe those Eyes create that future
It's similar to Usopp's Case, does He predict the Future when He lie? or his Lies change the future?
This is why imo Loki is considered Accursed & his Eyes are Covered, and why They believe He is a Doom Bringer