Katakuri was the #2 in a decades-long criminal enterprise that preyed on the vulnerable, innocent or otherwise. He was party to untold theft, murder, abuse, exploitation, slavery, terror - all of it. What he did as a child doesn't count, but he shares in the culpability for every crime Big Mom committed after he was old enough to know better and walk away.
That he cared for his siblings and observed combatant's honour doesn't erase any of this. I don't Katakuri would be so deluded to deny any of this.