Ya, that's my opinion too. Idk, I can't see all of them being losses between Neko, Inu, and Raizo, but I feel like Inu is the most likely to go down if that's the case.
I "could" see Nami assisting Raizo with Fukurokuju, same with Usopp helping Inu with Jack. All 6 of those characters are on the 2nd floor of the mansion. I'm not sure if that's on purpose or not lol (technically Hawkins/Killer/BM/Kid/Law as well).