Yeah he very specifically calls dragon "luffys father" and calls Luffy "the son of my child", he never actually specifically implies dragon is his own son.
If luffys mother was the connection here and Dragon married her, then something happened to her, it would neatly explain everything about why Garp hates the celestial dragons so much and everyone in the Marines seems to completely understand why and allow him to be that way, and why Dragon is so obsessed with taking them down completely. She was likely killed or enslaved or some similarly horrible thing and that was the last straw for them both.
Also I feel like this would explain Woop Slaps enigmatic "is it his dream or his destiny" line. Because it doesn't make any sense why luffys destiny would be to become a pirate. Dragons not a pirate, Garp isn't, Shanks is just some guy Luffy knows. There's no piracy at all in his family history or his past, unless... If Rocks was his other grandfather it would make sense.
ALSO I think it'd be interesting for Luffy to inherit Rocks' blood and have it not really matter to him, while Blackbeard inherits Rocks' will