Oda clearly created Kaido and Big Mom around Enies Lobby times. From there he starts putting hints about them. They were never referenced before that.
For Xebec is probably the same,and maybe for Imu too,they werent in the original plans.
For "filler" i mean: they are not "technincally filler" now,as Oda put big events in their arcs like the Gear 5 for example,and put them in the Rocks crew etc. But,the point is: if One Piece was still succesful but not as succesful as it is,and Oda had to end One Piece a few years earlier than how he is doing now,he would have never added Kaido or Big Mom in the story,but surely characters like Shanks,Mihawk or Dragon would be there anyway.