I doubt he's going to not have luffy be pushed to the extreme with the final, actual fights.
I consider what's next to come similarly to pre TS handling of villains. Lucci wasn't necessary stronger than Enl, but he was more dangerous to Luffy. If you don't have haki, Crocodile absolutely demolishes certain enemies.
What I mean by this is future enemies will play roles on what Luffy can't do. If he doesn't have FS mastery (never confirmed), then Kizaru is a pretty great matchup as a lightspeed opponent.
Akainu may have the highest AP in the series, but nowhere near as tanky as Kaido. He'd actually dodge and not eat 90% of Luffys attacks.
Blackbeard may have a 3rd fruit and the combination might be entirely overwhelming for Luffy because it'd negate Gear 5 completely.
Shanks, if an enemy, may have higher haki standards than Luffy considering his latest display.
Then you have the Gorosei and Im who are currently unknown factors.
Or like
@Seth said, maybe there's just no fucking logic and Luffy wipes everyone (hopefully not). I'm just saying the potential is there and we have to wait and see