He lives there, it doesn't mean he can leave the place whenever he wants to.
There are a lot of "ifs" here, where was Gaban when Shanks came by to begin with? Did Loki waited for Shanks to start a riot? Who was maintaining him under control until Shanks showed up to help beat him? (if that was even necessary, which is another "if" here)?
There are so many possible questions but all you can say is "omg he admitted Loki shits on him" when he is literally warning and lowkey testing Luffy to be prepared for whatever the heck he does there.
The saying "Don't stay in Elbaph for too long" exists for a reason that we don't even know yet.