That's exactly what I was saying. They couldn't hold onto that power and influence without Whitebeards strength. Doesn't matter how vast your army is, or how far your reach goes, without the overwhelming strength of the Captain, one cannot be a Yonko. As a matter of fact, Doflamingo's forces were bigger than Whitebeards, and one can argue Doflamingo had one of, if not the most political power and influence out of all Pirates due to his dealings in the Underground, and him being a former Celestial Dragon, but he wasn't even a contender for the title of a Yonko when Whitebeard died.