That's what hurts the most lmao
After Dragon and Luffy fucked him over Aokiji was the only one close to him. He also went rogue so he must have developed deep attachment for Koby as he is his last hope and he had to protect Koby at the cost of his life.
As much as people in One Piece world admire Garp as a hero (I mean Marines), Garp is still at the end of the day very lonely as all his loved ones keep leaving him and going their own ways. Very Tragic if you look deep into it.