Kaido's horns look NOTHING like Harald's. If they were to be related, they'd be very distant cousins at best. Also, Harald doesnt look like the type to reproduce with anyone other than giants so Kaido being an oni (a full one at that) makes no sense
Linlin banged a Tontatta, and her kid, Normande, is the size of a regular Tontatta, despite Linlin's size/genetics.
Kaido talked about Oni a decent amount while fighting Yamato, and we only meet Kaido when he's 10 in his flashback.
IMO, Kaido was clearly raised by Oni. Yamato is known as the Onihime, or Oni Princess. This leads me to believe there is an Oni Queen.
Harald was trying to bridge he gap between different tribes. Since Ancient Giants clearly have Oni genetics, it makes sense Harald wanted to meet with Oni.
Kaido inherited his Oni Queen Mother's horns, not Harald's (same with size).
IMO Kaido's Mom was an Oni around Ida's size (large for an Oni), and was killed by her own people because they feared Kaido's power, having Harald's genetics.
Also think Kaido grew up on a Sky Island, and the Oni had to fight each other for what food they could manage. Kaido slaughtered the Oni after they killed his Mother, then jumped off the Sky Island to try to kill himself, and landed in Vodka Kingdom.