Roger's greatest enemy was the one pushing around all the yonko that came after him.
Kaido was old enough and an active pirate during Roger's era
This is what the narrator says about Roger and Whitebeard:
The Pirate King is the standard that the WSM is compared to, not the other way around.