I definitely think he has serious issues with his own mother based on how he treats mothers in OP especially Big Mom too and him supposedly saying mothers are the death of adventure in the past or something like that.
It's just my hunch and might be Nonsense but it does feel like there's something personal with him there.
Too tired atm to research and elaborate , I'm about ready to faint like Sanji but luckily I'm in bed though lmao.