Actually, I think I have a better idea why Harald was giving Shanks the side eye. Like how we consider Chapter 0 canon, but not Strong World, if we don't take RED canon, but we do it's supplementary materials, Shanks was worth a hair over a billion by the time he went to Foosha.
So it may just be Harald remembering Shanks has a huge bounty on him and wondering why he's allowed to walk around freely.