But that's situational, you see? It's not Luffy who pursues the admirals; it's the admirals who pursue Luffy. Luffy's goal is to take down the Emperors, but Luffy has already established that he intends to be strong enough to defeat whoever stands in his way. However, his true goal is indeed to bring down the Emperors, as he mentioned in Punk Hazard.
So the probability of Luffy getting involved with the Yonkous previously was higher than Luffy coming into conflict with an admiral, given that he had the Emperors as his goal. After all, Luffy also believes that the Emperors are the most frightening things in the world – if you don't fear them, you shouldn't fear anything else in this world.