They are most likely the off springs of an ancient race, some oni giant tribe. Same like Oars and probably Kaido himself. Perhaps there are a bunch of cross breeds inbetween, which could explain those height differences. I could see them having an in group hierarchy based on height btw. Would be funny if they are roasting each other, cause of those height differences.
I already like it that they lose their manners when they are drunk. That's certainly something that can be exploited on in a funny way aswell.
I think it's obvious by now that I think that most of the same will be quirky guys, more funny, prolly a little dumb aswell, but scary when they get serious. Wouldn't be against a certain brotherhood vibe tho.
And I don't think that they are living in Onigashima. It might be that they were born there tho. It seemed like they were summoned for the fire festival afterall.