Exactly.
If Luffy had stronger CoC, he could have beaten the Gorosei. But, the Gorosei have CoC, too; Luffy needed to over-ride both their CoC, and the Abyss Regen with his attacks, and he lacked the Haki to do so.
Joy Boy has Haki that probably surpassed Roger's, so, Joy Boy's Haki could shut down the Gorosei's Summoning Magic that Saturn used, and knocked Saturn back to his ship, and the rest of the Gorosei back to Marijoa.
Since Gunko doesn't have CoC herself, but is having Imu's CoC Infused into her, Luffy/Zoro should be able to fight her.
Since Loki was injured by Gunko, and now Imu is possessing Gunko, I can see Loki getting a good hit off, too.
People hate on Haki, but, without it, this entire series would just be a Devil Fruit vs Seastone/Water festival.