The other problem is that Ulti and Page One are very loyal towards Kaido and because Kaido raised them since they were kids, they also see him as their father figure.
If Kaido doesn't die or gets imprisoned and simply just leaves Wano, then I do think Ulti and Page One will join him.