Ask yourself, in what other context would the Gorosei accept meeting a pirate (who is hiding his identity with a cloak from the public) and vice versa, where "good person" Shanks is okay with only talking to genocidal murderers who's actions are publically known (Ohara, Slavery, Racism, etc).
The meeting itself is weird, thats the point.