To me, this is clearly stating the reason Gyaban didn't go to Wano, is because of Kaido.
I do think it's more Oden/Joy Boy prophecy based than Gyaban being scared, though.
If the opening of the borders didn't need to happen later, I think the former Roger Pirates would have freed Wano years ago.
Oden probably asked them to not interfere and trust Joy Boy's vision.
But, yeah, Gyaban is smart, he knows he can't just visit Wano by himself when Kaido's ruling the territory. He'd be asking for a fight, and he'd lose.