Like, there are those bandits, theyre dangerous, theres many and Matahachi is like " well, there's these dangerous guys and I also want to become a capable swordsman, so what am I gonna do? Right, I'll let Musashi fight them all on his own while I fuck that one girl. Right. I'm gonna do this instead of helping my best-infact only- friend."
So he goes on having sex with that girl while Musashi has to battle all these guys by himself, Matahachi runs off with these girls, he even THINKS THAT MUSASHI DIED and feel oh so bad about it but theres nothing that can be done, he just accepts it and continues his life of being a pathetic coward, fraud and crybaby.