But even you admit that outside of these special VAs
1. Luffys rival for decades
2. The second strongest of a warlord crew
Every other VA has gone down like a chump.
Garp is a certified top top tier (or at least was)
In Marineford, Onigumo, Dalmatian and Momonga put on a fairly impressive showing. Their feats don't put him on the same level necessarily, but they were able to get involved in a fight between Admirals and Whitebeard's strongest commanders and didn't look out of place. We also then by portayal had Doberman, Strawberry and Stainless grouped with them in certain scenes
We also know that Kuzan, Sakazuki and Borsalino were monstrous fighters even 20 years younger as vice admirals
We also know Tsuru was/is a fearsome figure for Pirates to come across
These also aren't 'special exceptions', i've just talked about 11 people who have been or are Vice Admiral (then plus Smoker, Momousagi, Chaton, Vergo for a total of 15) out of a total of 37 known Vice Admirals - it seems like a good chunk are very competent fighters.