Probably something happened that we don't know.None of the stuff you guys mentioned fit the bill.
We dont know about the timeline.
When and where Dory and Brogy last met Shanks?
When did they arrive on Little Island?
What was Shanks' age when he met them?
Did Shanks meet them when he was on Roger's ship?
I think they met Shanks when he was with Roger.
Later, both of them faced some tragedy and thought they were dead.
Possible evidence;
Dory and Brogy didnt recognize Luffy's strawhat.
People who are close to Shanks know about the SH.
Or, it is just one more of Oda's inconsistency. Like, he changed Tama's age to align it with Ace's flashback.