That's not how his power works, he has to switch things. Putting a soul in another body means he has to put the other soul in the other body.
The point of BB's DF is that it can absorb everything, so maybe the downside is that Yami Yami users' soul is absorbed and trapped into the DF once they die, then their soul goes into the next user's body after they ate the DF. That would mean the original owner of the Yami Yami's soul was trapped into Xebec's body, and BB has that soul + his father's trapped into his own body.