No one but Luffy in the alliance could probably challenge an Admiral in a solo fight. GB lolstomping the alliance without Luffy is a real possibility.
And Luffy isnt challenging GB, Luffy is beating GB if they fight. Itll be a high/very high diff fight though.