That's why I find it so damn funny when people are like "THE GOROSEI SHIT ON THE YONKO!!! THE YONKO ARE TRASH!!!" because Kaido was literally chilling in Wano with his crew with delicious food, games, hookers etc. while plotting against the World Government for decades and they never tried to take him down at all.
But yeah, anyway, I highly doubt they could have went after Luffy after the timeskip. He already had Rayleigh, Shanks, Whitebeard's goons, etc. supporting him.
As to why they didn't try harder to take him out in the earlier parts of the series is a mystery though. Like, they had CP9 fight him, but that clearly didn't work.
And for some reason the only person hellbent on killing him was Sakazuki in Marineford, nobody else really seemed to care back then.