In most cases, those characters come to them. Rookies and such would rather deal with the likes of the Yonko than have an Admiral in their business.
That being said, fighting an Admiral has dire consequences. Everyone watched a foolish Commander getting killed by an Admiral on worldwide TV…and who wants that for themselves? It’s better going after prey like Yonko to take their territories and stay alive even in the face of failure.
Besides all that, there’s just the meta reason that the Admirals have to come later because they have tougher abilities for Oda to justify opponents leaving unscathed.
Sorry for being a bit serious about what was likely just a joke…