The Tobi Roppo don't know what the SHs look like from what Law told Sanji.
I think, only the ones that met them before (Hawkins and Drake) do. So apparently, they just had an encounter and said some rude words and then went their separate ways? So that Luffy, Zoro, Kid and Killer won't fight Ulti and P1 after all?