He had worse potential than Luffy from the beginning. Luffy was much stronger than Ace at the same age. That should tell you elready the gulf in talent between them. Even with an OP devil fruit, Ace still sucked.
His Willpower was small, that is what it takes to get to the top.