Unless you have a special race, your genetics do not give you a straight power up. You can say that somebody might have higher potential but then that should be true for every single person of that clan etc. Just like not every Shimotsuki has Ryuma potential, just like not every Gum Gum user has Joyboy potential similarly not every D member has Roger, Joyboy, Luffy potential.