I guess it's about the stakes involved.
If you go against a yonkou then it's against one pirate crew.. no other yonkou would jump in against you
But if you go against an admiral then whole fuckin navy is coming at you which includes 2 more admirals, fleet admiral. Also, they have more human resources to back them up.