What feats does Vergo have to indicate he'd require G4 to lose?
He took Sanji's DJ to his face, twice. He easily struggled off Law's Countershock - one of his strongest attacks back then - broke Sanji's fibula with his non hardening leg. Before you say something, that isn't even Vergo's main fighting style, he's best at using his bamboo.
We can say G2 attacks won't do much to Vergo, he'll eat them quite easily. Sanji hits way harder with DJ and you can basically scale it like that: G3 >= Sanji's DJ (depends on the attack) > G2 (maybe bar Red Hawk).
Before Luffy received his next level Haki PUs, base Luffy really was weak af except great durability. Almost all of his enemies easily overpowered him. Heck, Luffy couldn't even best a single biscuit soldier. Katakuri mocked him and copying his strongest moves but stronger. Doflamingo straight up called G2 weak - he wouldn't have this reaction to Vergo's Oni Take.
So again, how do you see Vergo losing to this? Do you not think you're underrating Vergo too much or do you believe he has to be weaker than other opponents because he appeared earlier as a villain? This kind of thinking is flawed in One Piece because what Oda rather focuses at is how the characters are tied and related to the story and world. Vergo wasn't only a VA, he is one of the strongest while being secretly a spy - working for Doflamingo as his right hand man.
Furthermore, Zoro didn't take someone like Pica seriously at all - something which Zoro wouldn't do to Sanji either if he was seriously fighting him. Yet Vergo managed to break Sanji's strongest weapon - his legs. It's clear Vergo isn't an ordinary VA or vet level character. He's above that stuff.
People just need to pay extra attention to the story instead of making up bullshit powerscaling rules like "The next villain is stronger."
Heck, you guys don't like it when it's being used against Kaido and almost all other top tiers being paired against him, so keep the same energy for Doflamingo and especially Vergo because the latter was truly a freaking beast.