Doffy not croaking from busting a gut at one of his underlings dying is not the same as him having the stones to actually roll up to a stronger persons territory for them and risk his own ass.
Doffy, respectfully, is a BUM with GREAT propaganda and aura. Straight up the ONLY warlord (bar Buggy I guess) that became another Pirates bitch in the form of Kaido. Moria (arguably) wasnt shown to be as scared of Kaido as Doffy was despite his crew literally being killed by Kaido and being mentally scared by him for crying out loud!
If you think for a second Doffy would ever risk his own skin for Vergo, or anybody of his "family" if they got snatched up by a Yonko and it meant he had to enter said Yonkos territory you and I have very different perspective of the guy, sorry to say.
Doffy aint brave. He ain't strong. Hell, you can argue he is cool/smart/whatever, wont hear me saying he has zero redeeming qualities (all the warlords have SOME of them) but he ain't this secret powerhouse people make him out to be. THIS guy aint the goat people claim him to be!