I don't believe Gaban waiting on Elbaf, he could still show up somewhere else. Because Rayleigh stated to the straw hats pre-time skip that he doesn't know where all the comrades went, with whom they risked their lives, or what they are doing right now. He was even surprised to hear that Crocus is still kicking. We also know that Shanks met Rayleigh on sabaody and told him about Luffy, so if Shanks territory was Elbaf all this time, he would have told Rayleigh about Gaban as well, which he didn't. Unless Gaban came to elbaf after the time skip.