They fight Shanks for a bit, only to throw a party shortly after like Roger Pirates vs WBP
Or Shanks outright beats Luffy - too early to return the hat, he's called Straw Hat Luffy, Straw Hat Pirates, so their next meeting will either be Shanks grave or Shanks fights Luffy in Laughtale
Shanks used his CoC to lure Luffy after him, to the 4th road poneglyph location. He said it's time to go after OP, Luffy has the other 3, and the 4th one revealed by Shanks, wait for the opportunity for Robin to decipher them and steal them from Luffy