Not necessarily.
Humans are complicated creatures and human relations are complex matters. The world is filler with "good people" that are dicks to some, as much as there are "dicks" who are wholesome and caring towards a few.
No single person is perfect and is loved or hated by everyone.
People also express themselves in different ways.
People perceive affection in different ways as well.
You have all the right in the world to think that you wouldn't like to have someone like Sanji around, because you would feel distressed by the way he treats some that are supposed to be his friends. But there others that don't feel the same way.
In the end, One Piece is a fictional work and all its characters dynamics are dictated by the mind of one man.
If Oda takes the time to draw an entire arc to say that "sanji is kind" and have multiple members of the crew reinforcing this idea, who are we to judge how these characters should or should not feel towards him?
Purge yourself from such hatred.
:P