Those guys aren't beating any Tobi Roppo or Ministers...... The Tobi Roppo and Ministers are tougher and have great devil fruit abilities.
Minister isn't a rank within the crew, it's a political position. Perospero is almost without a doubt not a member of the crew anymore (he's still loyal and would fight for big mom), and he's a minister still.
For example, Katakuri is a minister, but so were people like Saint-Marc, Dacquiose, Kanten, Marnier, Citron, Noisette, Zuccoto, Cornstarch, Galette, Broye, Nusstorte, Compo, Flampe, Chiboust, Kato, Tablet, and Praline. Honestly, if you're a big mom kid who is older than the age of 17, you're pretty much a minister.
I have no problem viewing Vergo as high tobi roppo level, and Trebol, Diamante, and Pica as Headliner leveled.
Trebol literally speed blitzed across a whole port in the time robin's clone (ie robin's normal speed) took to take like 3-4 running strides. After doing it to talk to others
Trebol lifted up a huge merchant and flung it fairly casually
Sniped a fly out of the air (and the power of the attack):
Exploding the giant ship:
Captured a base Luffy:
https://imgur.com/a/bb5WKD7
I have no problem believing honestly that Trebol could beat people like Counter up, and relatively speaking, he's decent among the "ministers"
Diamante is hard to gauge cuz honestly, his devil fruit isn't flashy, but he scales above the likes of gladius and so forth easily.
Anyway big mom pirates:
Katakuri > Cracker = Streusen > Perospero > Smoothie > Compote > Snack > Daifuku and Oven > Raisin > Tamago > Etc
Beast Pirates:
King > Queen > Jack > Doflamingo (unofficial member as joker) > Apoo > Sasaki > Who's Who > Black Maria > Ulti > Page One > Headliners > Numbers > Gifters > Pleasures > Waiters
Vergo would be around Snack leveled, Trebol, Diamante, and Pica more akin to Headliners.