Kaido isn't being defeated by just anyone, it's the protagonist on a series that is near the end and will become PK, PLOT wise Enies Lobby was also portrayed in its day as a place that had never been breached and one of the most important facilities of the government and besides a bounty increase there were barely consequences.
I really don't know If I need to explane basics of what good story is, but apparently I need to to do this.
Let's start with comparison of different stories and how they was build up:
1. Naruto was strongest case he has sealed Nine-fox demon in himself, that destroyed village and it took life of "strongest" fourth kage to just seal him. No one was ever asking question, why Naruto was strong enough to defeate anyone;
2. Bleach, it's a bit different...but Ichigo gained power to defeate Aizen with big pay-off losing his power, plus he gained shinigami+hollow power, which was already explaned as stronger than usual shinigami, so no questions there, and pay-off was huge;
3. Hunter x Hunter - Gon was apreccicated and "seen" by everyone by his potential, so in the end to defeate Pitou he used all his potential, and was nearly dead after that, losed all his nen;
4. Dragon Ball - in that story Goku was never even showed as normal human being, he was from different race, race of strongest, so there was no Big Deal to him became stronger than any other sayajin, case unlike others he learned not only brutal way of life, but learned also to love and care about others, which made him special between his race;
5. My Hero Akademia - Deku received the Power of Strongest...so how he can not be in the end strongest?!
6. Natsu was special from the very begining, he was END and much more, so there is no questions about him;
I can continue that list but thing is that all those shonen characters have power and that power was explaned, unlike One Piece where main theme was " make your dream possible, even if it is impossible". So Luffy was basically compared to others like just a guy with dream, that has no super power inside of him. And even learning COA COC did not made him different from others. But aside of him we have Ace, the Roger's son, who failed in defeating Younko. We have many other pirates whose life was destroyed because they wasn't able defeate a Younko: Doffy, Crocodile, Moria...all of them, and much more people. Because of that fact only there was build whole plot lines, that made em live peaceful life, plotting their revenge like Crocodile. And we talk about people who was adults, not just youngest of them. Ace was young but he failed in his mission. BlackBeard planned whole his life to became stronger then Younko. This is not about power scalling, this is about PLOT of ONE PIECE. Now question is what they lack, that Luffy has?
If Oda made just Luffy super power guy who is special, it wouldn't be big deal right here, but it is case Luffy just success in something where everyone failed.
And that is the problem,NO explanation, NO excuse...this is just FAIL. EPIC FAIL.