I don't think Rayleigh can defeat her outright, but he's much smarter and could trick her into losing.
If they're fighting 1v1 inland, I give Big Mom the victory. Her defenses are too high, if you can't hit her internally/disrupt her mental state, you just can't harm Big Mom. If they're near a deep body of water, though, I trust Rayleigh to be smart enough to knock Big Mom into the water and defeat her.