Just looking at this scene in isolation ultimately suggests he is not "really" on Luffy's side:
Like theres pure antagonism drawn on his face lol, you dont draw characters lkke this unless you are trying to paint a picture about them, albeit loosely.
I dont even need to mention his Gorosei meeting or my PFP or how he absolutely brutalized Kid