Losing willpower is the biggest difference there is.
Willpower is what keeps one going and why rookie with unbroken willpower outlasts veterans with broken willpower.
Rayleigh's willpower was never affected due to inactivity.
He didnt suffer a dream-ending defeat like Crocodile did.
How much can Luffy last during his phase of broken willpower after Ace's death? Not even a competition.