Tbh I don't see the Yamato thing as a shocker at all, like Yamato did not meet half the crew until the end of the raid, and had next to no dynamic with the crew, even as a villain Robin had interactions with all the Straw Hats, then only he made claims of wanting to join the crew, we never saw Luffy say nothing about it, and even in the end there was always something that felt off about him joining. All in all I don't see the big fuss over this, his reasoning also makes sense since he was in chains since he was a kid in Onigashima, never got to visit Wano, so seems fair to me