Even if he still is, your idea could still be right, just... not Luffys father is Xebecs child, its his mother.
I mean, just in case you are wrong with dragon, but right with your idea at all, it could mean Xebec had a daugther, which may be was taken home by garp after beating Xebec, so...