Let’s review things:
Garling left his son, Shanks, in a treasure chest, and Roger found him on God Valley. Garling used to be the King. The Rocks Pirates suffered a significant loss there. They obviously invaded for some reason. The question is, who did Rocks lose against? Was it Roger and Garp, or did Roger and Garp just take care of them when Rocks were weakened leftovers?
Garling is definitely strong since he’s the head of the Holy Knights. Why would he need to leave? If he fled, he wouldn’t leave Shanks in a box unless it was his mother who left him in the box.
The only chain of events where all this makes sense is as follows:
Imu decided to create an order of Holy Knights for whatever purpose they were meant to serve.
Perhaps it was for defense against Rocks and Roger back then, so Marijoa could be protected.
Garling left; therefore, the king, the strongest in the kingdom, wasn’t there to protect it anymore.
Xebec decided the best way to inflict emotional damage was if he couldn’t attack Marijoa directly.
Shanks’ mother hid Shanks in a chest.
After a long battle, the weakened Rocks Pirates encountered Roger and Garp who were having a little family dispute and then decided to turn on the Rocks Pirates instead.
Would there be a reason for the Gorosei to be present here? One can assume yes because why else would there be a flashback now after the Saturn full reveal? Does one of the Gorosei inflict some kind of illness on people? They do seem to cast curses. Which is where Roger might have gotten his sickness from.