I mean, miss Doblefinger was introduced as Mr One's partner, and Kalifa was the only CP9 agent to be introduced in W7 alongside Lucci, Blueno, and Kaku: Luffy and Zoro's opponents.
Given his role as a chief of staff of Blackbeard pirates and being portrayed as BB's most trusted man, it is more likely for him to fight Nami. If Ussop and Chopper were to fight Augur and Doc Q, what makes you think the navigator of the BB pirates fights the cook instead of you know, the navigatress.