It's a guess. We have no idea what their actual relationship is. When Garp has met Luffy, it's casual anger/love, but when he talks about him separately, it's either nonchalant or an ironic proud moment lol.
With Dragon, yes, it's nonchalant, but I'm saying maybe they do not actually like each other. I'm talking what might happen if they do actually meet. What would they talk about?