I do think Doffy knowing about Imu's secret was a big deal at the time Doffy found out about it in universe, and in the story(like Dressrosa), but we're moving into the point of the story that people will know about Imu. We don't know exactly what the treasure of Mariejois is yet, but since Luffy is destined to fight him again and find out everything on Laughtale, it can be covered in the final war/Mariejois arc. Luffy isn't going to want Doffy to tell it to him the same way Luffy doesn't want Rayleigh to tell him if the OP is even real.
Besides Crocodile and Buggy, all the pre TS villains who returned to the story were never imprisoned(though Buggy wasn't imprisoned after losing to Luffy, it was in some cover story), Oda chose to have Lucci escape w7 and a cover story about him, while Moria escaping was made clear at the end of MF. Tbh, some of those cases don't even really make sense to me. Moria legitimately hasn't done anything important since, so I wouldn't be shocked if those guys aren't being kept around to be hypetooled in some way(yes, this includes Crocodile, btw). I guess it depends on if you believe that Luffy will do another Impel Down breakout, but that feels redundant, and Luffy has way stronger allies now.
Ironically, if Oda wanted to bring back a bunch of villains to be generic Luffy allies, it would be like, King,Queen, Katakuri etc. Technically they're all as "evil" as Doffy, but since they're just henchman, Oda doesn't take it as seriously, and it's not clear they were ever imprisoned(I know some people think King and Queen were captured by GB, but I don't think so)