It does matter and Rafa has two golds (1 singles and 1 doubles) and Roger has 1 (doubles). An Olympic gold is still an Olympic gold.
I'll say this tho, Novak does have the edge over Rafa and Roger but I am sorry what Laver did was out of this world. Saying that, if I needed a player to play for...