Zoro what's the first person to make him start huffing. He's literally telling us in that scene that Luffy disappointed him.
After Zoro scared him Kaido stated he could have ruled the world with them. That was before Luffy showed his in battle growth.
This scene tells us Kaido feels like Luffy wasted his time that's why he said he should have just killed Luffy and announced victory.
You can't honestly believe he was holding back when he used a combination attack with Big Mom that was stronger than any individual attack he has performed.