I get that, but Judge hated Sanji specifically rather than Yonji Ichiji or Niji because he was human
Yes, that was a part of it.
After sanji grew up a bit his brothers started to manifest abnormalities (abilities), while sanji showed emotions he should be lackin.
Judge listened to one of the sientists suggestion to give sanji more time to develop.
After sira's death the brothers started developing exoskeleton and superhuman strength, while sanji was lagging behined in the scores, that's when judge considered him a wast of life due to a total lack of future premise.
The brothers didn't start bullying him untill they were old enough to begin training.