I know that but they were in the same crew how did the location not slip out after?
They probably do know the location, maybe not. Depends on how they feel about it. Luffy could have also known by asking Rayleigh but preferred to find out on his own, maybe Shanks is the same way.
After the Roger crew returned from Laugh Tale, they picked up Shanks and Buggy and then everyone went their own way because the Roger Pirates disbanded. Shanks said he wanted to go with his own crew so he might have done so already, we'll probably never know until Shanks talks with Luffy.
Buggy on the other hand was probably too weak to go through the grandline/new world and reach Laugh Tale as an adult to find out personally what the One Piece. So I doubt he's gone even if he knew where it was. He might know what it is though. I doubt he wouldn't have asked the crew when they returned.