So this chapter confirms that there is no time left for Loki’s conversation, and that these groups will not clash because Luffy, Nami, Zoro, and Shiki are too far away to make it there in reasonable time to make a difference.
Is it possible that the conversation with Loki and the current castle situation aren’t happening at exactly the same time?