It's not that Ryokugyu is particularly handsome, it's just that compared to the rest of the admirals and Yonko, he edges out. And note that we are still talking about like fifth place and not the top.
Kaido also has a great design. He looks like a Mongolian type warlord, like a fierce beast. In that regard, I would place him above most others, maybe even #1. But we aren't talking about overall design here, just about handsomeness and here I think more women would choose to fuck with Aramaki as opposed to Kaido. As in, yeah, Kaido will get pussy in his home turf, as the leader of the beast pirates, but if he walked into a bar, women would probably be more terrified seeing a 23'4 horned Oni walk in as opposed to an ordinary guy.