But the presentable public image is maintaned here through the erasure of God Valley from history. We as readers know better of course, but not every person in One Piece has this information immediately available.
This is why Roger and Garp get judged even though they have no idea of WG plans or what is going on at the island.
And in the end, Garp considers CDs as scum as he himself said, but his moral compass has probably decided that there is evil and less evil. Still too early to judge since we are missing a lot of context.
In real life, we have rational, good people who defend cartoonishly evil people. There will always be justifications for this.