Koby's dream = to become an Admiral
Dragon = wasn't strong enough to climb the ladder to the admiral or fleet admiral, otherwise he could've changed the marines and started revolution from the inside out
Garp = doesn't have enough balls to resist orders from Celestial dragons/WG, so he's afraid to be Admiral rank, so he rather stay VA.
Meanwhile Fujitora straight up can oppose Celestial Dragons/WG while being an Admiral.
All of them can be claimed by Admiral faction, it's just they weren't strong enough/willed enough to be an Admiral.