All that in fanficyion
In One Piece :
Rayleigh isn't yonko level, and couldn't help RP beat any rival crew
Roger isn't not Kaido/BM /WB with broken fruits and durability
He's not stomping any yonko crew , he might win extreme diff
Roger just wanted to avoid casualties, his military power at the time is easily superior to Big Mom's gang, if we assume that it had the same current dynamic, where its crew members are his children.
At the time Roger stole his Poneglyph his children were either very young or not yet born. Rayleigh was at least strong enough to pose real danger to Teach Yonko, given both characters' statements.
In One Piece a single outlier character can decimate entire armies, ignoring the difference in numbers.
What resources did Big Mom have to counter this?
Considering that she would probably come into direct confrontation with Roger.
If you imagine a conflict scenario, where the Roger pirates can exercise their notion of power unhindered, an indisputable victory will be evident.