Eh partner and brother aren’t the same. If they were partners Oda would’ve used the same language he has for the others. This was very specific wording towards brother.
And Fuji is fighting Zoro if Sanji is fighting Ryokugyu. Oda tends to pair their opponents like King and Queen, Mr. 1 and Mr. 2, Kaku and Jabra. Ryo and Kizaru aren’t really a pair (plus Kizaru isn’t really much of a swordsman)