Hard to be well adjusted after being imprisoned for 20 years and having Ace as your only friend.
And Yamato isn't a isolated case either. We saw how hateful and distant Ace was because everyone just saw him as, Roger/ the devil's son.
We saw how of a split personality Pudding has, because of the long time she had to always put on a act for her mother Big Mom.