If Kizaru loses here and Fujitora defects, then the admirals aren't the final villains.
It's going to be gorosei and Imu. You guys might not like it. But that's literally how Oda structures EVERY arc.
Big Boss fights Luffy. His main subordinates then fight Luffy and maybe some allies.