Ngl, I can't blame them dipping after hearing about the story. What would you rather do, fight against a godly entity throwing a sun at you to save the world or banging a hot giant in a random village?
Roger definitely said "Yeah... I ain't doing shit! Someone else will do it... yeah, I'll just create a new generation and make all the idiots believe there's the greatest treasure in the last island... yep. I shall do that!"