If Gaban ~ Rayleigh, he should be solid Low/Mid Top Tier, and be able to fight Admirals to a high-extreme difficulty fight. Even if Luffy is using ACoC in base, Gaban should still be able to match or even surpass him.
Him clowning base Luffy with no ACoC is no surprise, and even expected. If it's Boundman Luffy, that'd really be a feat though.