We know Harald used to travel a lot, the timeline fits. Considering Kaido has attack named after ragnarok and the three worlds, his horns, Loki using a kanabo as a kid I feel the odds are pretty high.
Oni doesn't seem to actually exist as a race, Kaido just called himself that because he didn't know he's half ancient giant/half-human imo
Harald was bridging the gaps between nations/species, too. Obviously, Harald would be interested in his Oni genetics. We know from Kaido that some Whitebeard sized Oni exist.
If Harald produced a normal Giant by mating with a normal Giant, then an Ancient Oni Giant with a normal Giant (though Ida isn't from Elbaph, and Estrida is, so latent genetics could factor into this), it's entirely possible that Harald mating with a Kaido sized Oni could produce... a Kaido sized Oni. Look at the size gaps in this series with Neptune and Otohime, Harald making love with a Kaido-sized woman is entirely possible.
Kaido went on a BIG rant to Yamato about Oni/Ogres being unable to live in peace with humans, because humans would fear their strength.
All of this points to the most common thing in One Piece: tragic mother death in character backstory. Kaido hates humans because humans killed his mother for being an Oni. This is how I'd guess it went:
Harald travels abroad more, and meets a new woman; an Oni woman who is barbaric, but fun/carefree. Harald falls in love, and, decides to have a fling, since Estrida is dead, and Hajrudin's mom possibly died, too.
Eventually, they make love, and Harald then has to leave. The Oni-Woman becomes pregnant, BUT, the WG has been spying on Harald because the idea of Elbaph allying with other nations terrifies them, it could seriously threaten the World Government's dominion.
Cipher Pol is ordered to kidnap the Oni-Woman and her Son in order to blackmail Harald. The Oni-Woman is told what is happening, and, knowing Harald loves her, and knows nothing of their child, kills herself so she can't be used as blackmail. After his mother dies, the baby goes into a rage, and destroys the Cipher Pol ship.
A few days later, a child washes up on the shores of the Vodka Kingdom.
Harald comes back eventually, and others on the island of the Oni-Woman tell him the WG kidnapped her. Furious, Harald contacts them, but, is told that, in order to get them back, he has to make a deal with the WG (the WG lies and pretends the Oni-Woman and Kaido are still alive).
Since Harald and the WG don't know what happened to him, Kaido is raised in the Vodka kingdom.
Eventually, Cipher Pol hears that there's this huge 15-year-old with horns wrecking shop in the Vodka Kingdom. Cipher Pol goes: "OH FUCK, HARALD'S ONI-CHILD IS STILL ALIVE" and send a ship to pick up Kaido.
Kaido is Kaido, and refuses to be kidnapped. Cipher Pol keeps sending people after Kaido, and he gets annoyed enough, that he seeks out Hachinosu for protection, thinking they won't attack him on Rocks' territory. There... Kaido meets Rocks...
The man Kaido and his half-brother, Loki, would both come to admire as their hero.
Years later, Kaido learns from Cipher Pol Agent Who's Who how his mother was killed, and vows eternal revenge against the WG.
Edit:
@Kurozumi Wiwi Do you think Oda is going to build up Kaido's character a bit here before we go back to Wano for Teach and Pluton?
Would be wild for the Loki flashback that turned into a Harald flashback to turn into a Kaido flashback
Oda's like "Why not 3 dead moms???" he's evolving holy shit
