Since Queen knows Judge, and Judge knows Vegapunk. It's highly likely Queen worked with Vegapunk as well. And we all know Vegapunk did experiments on Kaido....so maybe Queen met Kaido when he was Vegapunk's subject. Then he ended up freeing him and joined his crew.