Rhaegar was very mid on fighting. Good, but nothing exceptional. He was always more peace, poetry, etc kind of guy.
Then, out of nowhere, dude has some dream then decides to participate in a tournament, wins it, then dedicates the favor to Lyanna Stark.
Whether or not you consider his "potential", the fact is that Rhaegar was never mentioned as a top fighter except for this very specific incident. And it was just a joust tournament.
iirc.