Idk what to tell you. Oda should have tried putting them in the story earlier.
You actually know it's a retcon because Oda even went out of his way to change plots to make it make sense how they exist(Shanks helping fisher Tiger), though it still falls flat, since we're not sure the Hk weren't looking for Fisher Tiger.
Oh, the Shanks/Tiger thing definitely a retcon, and a very poor one at that. But no, I can't say Imu and the HK are a retcon for one simple detail. There was never a first among equals with the Gorosei.
Think about it like this. If all five are equal in power and rank, then who's the one who Luffy fights, and then by proxy, Zoro and Sanji? So, their scars indicated they are knowledgeable about combat, but we know how this series goes. Luffy the main leader, Zoro and Sanji, the right and left hand men.
Which means there had to be an ultimate singular foe for the WG. And by that token, proper underlings, like the Tobi Roppo to the Calamities.
Again, we can type to our fingers turn blue about timing, but Imu was definitely something planned out even as early as Marineford if you believe Doflamingo's statement about a higher power.
....Now, do I enjoy Team Mariejois as villains? Fuck no! But that's a whole different debate.