I was gonna reply to your big post about this earlier but you said it yourself.
Not saying trans people should restrooms or not, nor does this involve Photon, but the main motivator behind Trans people shouldn't be allowed to restrooms based on their preferred gender is the idea that they're sexual predators, and that argument obviously doesn't work.
Who says all trans women are sexuallt attracted to women in the first place? They could like men. On that note, should homosexuals be banned now with that logic? If you put them in their own restrooms, wouldn't assaults still happen? People can't pretend it isn't a factor seeing how homosexuals have been portrayed as sexual predators who can't control their sexual urges for decades.