How is that even a question? Every day he’d stare out towards the East Blue, thinking of Luffy. He obviously loves his son. Staying away from him is an example of that. He loves his son so much that he stayed away from him his entire life in order to protect him.