As much as I can slander Oda I refuse to believe he's this senile, GV flashback was like a few montes ago. They likely survived and Xebec assumed he killed them (cause he went for the kill). Maybe we'll learn they re part of the people Kuma saved.
Still, would be very cheap of Oda to make it seem like Xebec was forced to kill his family if he didn't. It's bad writing either way