It became much more evident that Kaido joined the Rocks Pirates by losing against Xebec during his apprentice days, with that loss being his first ever before his second loss on God Valley where he ate his devil fruit.
Unlike how Luffy recruits his crewmates, it seems that all the notorious pirates from the Rocks were actually defeated/subdued by Xebec, he created this powerful band by strength. That's definitely what inspired Kaido to break and subdue strong pirates like Luffy and Kid. It's completely opposite to Luffy's natural ability to make allies peacefully. Kid mocking Law for following Luffy's orders made me think about that, Law isn't forced to do so, he did it of his own free will.
That explains the violence and lack of camaraderie in the crew, and why Kaido always talk about concepts of "betrayal" and "treason" being the common qualities of a pirate. Oda might play a little with that theme to show how Luffy's way of recruiting allies wouldn't fall apart like Rocks did, and that's something Kaido will notice right before his death when the Worst Generation is ganging up on him.