Thinking about Yamato’s decision to stay again, maybe it shouldn’t be that surprising. At this point, Yamato does seem like someone who'd pick Oden and his path over Luffy and his path any day of the week.
As if to say, that Yamato basically sees Luffy with less relevance since Luffy isn't Oden, and that Oden has more relevance than Luffy does as far as Yamato is concerned. Maybe Yamato doesn't even see Luffy's journey as that much of a priority to get involved with either.