By the time news of Wano's fate reached the WB pirates the logical conclusion was to assume that all of Oden's family and scabbards were dead, they weren't there for Toki's words so the prophecy doesn't matter. Most of Oden's family and friends were either in hiding, literally missing from time or dead.
From WB's perspective:
-Oden returns to Wano, which due to it's closed off nature has it's events rarely told outside of it.
-Years later it's publicly known that Kaido has made Wano his base and that Oden is dead and since WB doesn't know about Toki's last moments, he and Izo probably had no choice but to accept the death of their friends.
-Attacking Wano at that point would have been purely out of revenge, and as much as you want to downplay Kaido's crew, it's still really fucking big so a full scale war would've broken out.
-WB is no Luffy, he had full awareness of his position in the political world of OP. A war between the WB and Beasts pirates by itself would have already endangered the people of Wano, while also attracting the attention of BM and the Marines.
All that said, it is entire possible that Oda had to retcon things or retroactively add context in a sloppy way, the man's not a god and One Piece's story is huge as it is.
To me some some strangest behaviors come from Inu, Neko and Izo.
Neko and Inu weren't at Oden's castle to actually see Toki send the others forward in time, so chances are their knowledge was similar to that of Ashura, Denjiro and Kawamatsu, only knowing of Toki's last words but having no actual proof that anything would happen until Raizo came back. Unlike the others however, Inu and Neko got to leave Wano and return to Zou, so it honestly surprises me that they apparently never met WB again.
Izo's behaviour is more strange in the sense that we never got a clear reason why he just decided to stay with WB and never help Oden in the first place.