Brook is completely irrelevant. I don't dislike him, but he is not a necessary crew member at all, and doesn't bring much to the table.
Jinbe, the crew went this long without him, & they'll be just fine if he leaves.
Chopper, but just because I personally find him annoying af!