I think they grew accumtiomed to the RS's protection from constant use and became complacent, nott expanding too much efforts to increase their own individual strength. Powerwise, bare Big Mom, they were born with the biggest potential out of all characters in One Piece.
Imagine them training in Rokushiki, developing their Haki, maybe even learning a martial art like black leg style or the one used by those from Kanokuni, forgot its name, developing their genetic power like how Sanji inhanced his DJ etc... Basically they'd be an unstoppable force. What a wasted potential.