Nah he would be, If Akainu was in place of Kaido right now, with Luffy's goal to beat him. And after having defeated the scabbards, he would absolutely do the same thing.
This is Luffy's story, not the admirals or Yonkos.
In the end he will beat them all.
Also no one is beating Primebeard or Roger 1vs1 but the cream of the crop
I.E Kaido/Akainu/Mihawk/BM can sure as hell give them a tough fight and nothing suggests otherwise.
And one last thing, stop underestimating Luffy. Its over, the man isnt a small fry to top tiers anymore.
Right about now i'd say he's around Old Rayleigh's lvl, AKA a low top tier.