That's what I think too but ...
The chapter literally says that Kaido and his crew does sailing ... so they're not always around Wano citizens !
Then, how come Whitebeard didn't track Kaido when he was sailing outside Wano ? He could very well have ambushed him on the sea route that leads to Wano without any casualties among the citizens ...
That's a HUGE PLOT HOLE, if this is Oda's way to explain why WB didn't attack Kaido