Right hand man argument, I guess.
Fuji/Ryo are newcomers whereas Kizaru has worked with Akainu the longest plus been an Admiral the longest. He also fought with Ray/Marco, two other right hands. So he’s perceived as the King of the Admirals and thus the better prize to fight.
Also a bit of a perception that the OG Admirals are stronger.