Re-reading Chapter 907, when we thought it was Shanks, it does make sense that the Gorosei were brusque with Shanks.
...but if it was Pranks, why the cold shoulder to a Holy Knight, saying it was odd he was involved with politics and how they usually wouldn't see someone like him?