It's really ridiculous when you think about.
They were willing to risk a World War just to kill the son of Roger, who hadn't even done anything yet.
They eliminated an entire island (Ohara) for simply researching the ponegliffs.
They were going full scorched earth on Vegapunk/Egghead, when he himself didn't even know half the story.
Yet Rayleigh, who actually knows the FULL story (is even willing to tell it like we've seen), is allowed to chill his nuts on Sabaody as their neighbor for decades.