Not necessarily weak, he is just the BY FAR worst fighter in the series of that level. Ceasar is similar, he has great abilities but is a terrible fighter.
Guys like Who's Who, Jack, Sai or Vergo might or might not win against Queen but they are 10 times the fighter he is.
It doesn't mean Queen or Ceasar are weak/strong as in; can they defeat 98 % of the OP universe? Yes!
It's like that big guy at school that never trained fighting but just has some abilities that allows him to beat most people but up against someone that knows how to fight he might lose.