Well, to be fair. It's speculated that Kaido is from Onigashima originally. And it's an island that can be seen from Wano itself. kaido himself was an apprentice. Meaning a NEW recruit. So, having Rox visiting the place is NOT farfetched. Specially if he visited Onigashima and recruited Kaido. Specially that Wano as a country is more prominent of an island to visit than Onigashima.
And if by chance the poneglyph that Kaido has in his possession now existed in Wano or Onigashima during that time long ago. Then, with Xebec looking into the taboo subjects may actually also prove of the idea that he visited/passed by Wano somehow during that period.