WW's beef with Luffy isn't serious enough to be a primary motivator but more of a painful reminder and plays fairly similar to Buggy's and his issues with prison might be related to the sadistic nature of Impel Down rather than being enslaved but we can't tell for now.
Yet people wonder why I sometimes lose faith in humans