In away I think judge has a heart and as so he give Sanji his invisibility on purpose because he most likely over heard him about his dream to go invisible, plus Sanji raid was more durable than his siblings showing that judge wanted to protect Sanji because he knew he was fragile when he was a kid, he wanted to protect him/ care for him in his own way even though he shows a cold side, just my Lil take I could be wrong but whatever