Unless you were a coping Zoro fan or Sanji fan or Admiral hater, it was obvious that Luffy would fight Kizaru again unless Kizaru did switch sides.
The only other way for Luffy and Kizaru to not fight was if the unknown party ended up being either Blackbeard, Dragon, or Shanks.
But the unknown party being Dorry and Brogy and Kizaru not switching sides, pretty much meant that Luffy vs Kizaru round 2 was going to happen.