Sanji doesn't believe that black people are competent fighters because of his nazi education
That's why he never fought any of them.
Meanwhile Zoro is intendly fighting black people in his path to become The Strongest because he knows they're among the strongest persons.
Why do you think his destiny is to make a black sword like the Akrafena (a "ghaneean" sword) ?