I think Mars at least might be more than meets the eye. Remember Ohara?
You can see he clearly looks tormented at giving the order to attack.
Oda seemingly retcons this later with all his dialogue of insects and such. Interesting how he was also alone when the other Gorobros reappeared.