Fair.
You're right, we don't know of any direct connections between Kidd and Elbaf.
There are number of things id like to mention.
1) Like I said, Shanks/Giants were the ones who destroyed Kidd, his crew and his ship..Victoria Punk. The anime even added a scene of him caressing his ship before the fight and you can even see his reaction to seeing killer down. Kidd will becoming back with one thing on his mind..revenge.
Look at his interaction with Kaido. Kaido landed right in front of him and Kidd/Killer attempted to fight Kaido to the death. The only reason Kidd is even alive today is because Kaido spared his life, similiar to Luffy. When Law met hawkins, hawkins himself, even stated that Kidd will die because he is never going to give up...
When sent to prison, at some point killer is going to be sent to the mines for reconditioning. Kidd learns this and immediately freees himself and tries to ambush the captors moving Kidd, however he doesn't fight back at all...He actually allows himself to be recaptured and tortured just be alongside Killer, aka a captain that will go down with his ship.
After escaping prison with his crew and Ship, Kidd eventually returns to fight Kaido. The crazy thing is Kidd was going to fight Kaido again with or without Luffy/Law, he even rejected Luffy's offer to teamup vs kaido (although this ended up happening anyway) and its likely he would have died...
Even later on, when the entire crew learns killer has the smile fruit, they all laugh to make the crew "joyful". Interestingly there are some other notable crew described as "joyful"....
I kind of went on a tangent, but my point is, its clear Kidd clearly has immense loyalty to his crew, will want to get revenge on anyone who wronged his crew and an undying will....It goes well with his fruit, no matter how broken something is...it can be rebuilt. In a way his fruit and Luffy's are opposites, rubber causes things to bounce/reflect and magnetism causes things to attract.
In this case he will 100% go back after the giants/Shanks. And we know the land of the giants is Elbaf. And I agree I don't see Kidd ever saving Elbaf, if anything he'll cause havoc.
2) The other interesting part of this dynamic is big mom's defeat. Big mom was clearly the giants most detested enemy. At this point the strawhats are viewed as the ones who brought down Big mom. They still have no idea Kidd dealt the final blow. Once, this is revealed, I feel this will be a huge point. Kidd will undoubtedly earn the respect of many giants once they learn this. It will also be interesting because Dorry/Broggy played a pivotal roll in his destruction. How will the giants react? At first, I didn't understand why Oda had kidd fight Big mom, but now I believe it was setup for Elbaf
3)Shanks/Luffy/Kidd dynamic. I've written a very long post before on why Shanks will be the catalycst for Luffy/Kidd to fight. For now all ill say is that, the relationship with Shanks between Luffy/Kidd is inverse. Shanks has played a large role in the destruction of his crew and Ship. I also personally believe he may have had some involvement in what happened to Victoria/gangs. And on the other hand Shanks has been everything for Luffy, his idol, his protector...someone he strives to reach/surpass...How will Kidd react once he learns how close Luffy has a relationship with the man, that was responsible for the destruction of his ship, crew, arm(yes I know it was Beckman)? Kidd will be enraged 100% and Luffy will not sit by and tolerate Shanks slander. I think the demaro black panel confirms this in a way. Luffy and Kidd will fight, I think Elbaf is a likely venue.
4) Lastly, the burned man. Kidd was the first character to mention the burned man. I believe the burned man is 100% at Elbaf. Kidd may even be with him right now...but honestly i don't really see anywhere else he could be outside of Elbaf.
There are some other theories like with Loki, rocks, acoc etc..but for now I think that covers most of the major ones.