By your reasoning WB, Shiki and Big Mom should not have been Rogers rivals as he beat their captain Xebec.
Zoro should not be able to beat Supernova as they were Luffys rivals yet he took on or out Killer, Apoo, Drake and Hawkins. Killer same with Hawkins. Oden almost took out Kaido who soon became Whitebeards and Big Moms rival, so strong WB declined going after him.
You also fail to take into account how crews and individuals may become stronger at different pace. Did Shiki keep getting stronger after Rocks? Who knows. Most likely not at the same pace as he would start to scheme instead of going into battle under the reckless Xebec. Rayleigh likely kept getting stronger until the day Roger Pirates disbanded and he retired.
