Then the Judge statements are irrelevant. Since Sanji is already fighting Queen for some other reason. Queen is not just gonna make Sanji fight him over a few statements about Judge.
Like, Sanji was targetting Absalom because he kidnapped Nami, but him eating the one Devil Fruit Sanji wanted to eat since he was a kid added to the conflict between them.