Well we don't know brookes childhood. Certainly it wasn't Nami, Robin or chopper. For Franky, Luffy and Jenbe they had brothers to grow up with. Comrades that they loved. Sanji and Usopp were relatively alone estranged from blood relatives although they did have friends they could run to even if it was unconventional.
Luffy, jenbe and Franky grew up in ghettos and that's hard but still a life. Brooke had a prestigious job before he became a pirate so he probably had a decent upbringing.
I still want to say Luffy because while he had it rough he was so simple minded and dumb he was probably happy every day. His only real tragedies is his mentor losing an arm and his brother disappearing presumably dead.
Technically he was still a kid when Ace died though.