First of all, he didn't have to endure any kind of bother from Baby 5, but he did. That already makes him pretty mild natured as a villain. It's not about her actually being able to hurt him or not, it's just allowing someone to flip out at him freely.
He likes seeing others hurt, but even in case of Monet, he didn't have to move a finger to do it himself. His strings, actions and business are a symbolic way of showing that Doffy is a manipulative character. He rarely does stuff himself, he doesn't dirty his fab royal hands etc.
It's not as black and white as you see it, Viola as a woman had a lot of mixed feelings, she got attached despite being forced to be part of the Doffy family. She wanted to kill him to make it right, not just because of her family/country but also because they got very personal. That's why she also has a pet name for him.
IF ONLY ODA COULD HAVE SHOWN IT PROPERLY