Some people say Sanji fans started them but people have said stuff like that about Oda many years ago on various forums, even at pre TS apparently.
There were rumours his wife is unfaithful with potentially multiple men and even Sanjis VA.
Its not impossible and given Oda barely sees his wife and kids especially when he was doing up to 21 hours a day regularly...
But if this is true them its very morbid, sad, dark, bitter etc for Oda lol.
Given how extremely hard and long Oda had been working for decades now, I doubt his wife gets to experience anything hard and long due to his extreme work ethics lol.
Cheating is very common IRL too unfortunately. The Western media and Internet has been trying to glorify it and encourage it lately top, for women to cheat especially and the whole "Ashley Madison" thing too.
Odas wife is a former cosplayer, theatre actress IIRC and gravure model amongst other things, so she's used to a LOT of attention too. Not something you can give up exactly, especially for marrriage/life and a single person who you barely ever get to see and spend time with too ironically.
Ive always felt he had an unexplained vicious streak towards Sanji and at one point at least his VA too and from the timeskip to WCI especially, I was extremely concerned by Odas behaviour towards them lol.
It might all just be nonsense though ofc.