Zoro was the only Straw Hat that Kizaru actually attacked. Usopp and Brook tried to hurt him, but Kizaru just ignored them to focus on Zoro. Ray stepped in after that. Kizaru as a weird tendency to face the #2 of various crews - Zoro, Ray, Marco, and Benn Beckman.
Fujitora had a 1v1 with Luffy and Sabo in Dressrosa as well. Plus Fuji is a Luffy fanboy, you really think he's gonna wind up as a true antagonist later in the series?