It didn't all happen on the 4th day:
It all happened after, otherwise it makes zero sense for Shirahoshi to not notice something wrong with Vivi until after she got back to FI.
I suggest rereading Chapter 925/926, when Sabo was shown to have fought the Admirals, and then 956, where you can see Fuji is already on a ship going to arrest Shichibukai. The "bad" thing that happened to Alabasta happened after everything