I mean that he couldn't control them as has been explicitly shown since day one:
—Shiki was laughing his ass off when Harald thought he was Rocks' subordinate.
—Linlin, as soon as he met Kaidou, told him he better followed her instead because Rocks was a loser.
—Whitebeard, far from accepting Rocks as his leader (which is what you seem to suggest with that "shut up"), instantly reminded him he wasn't his subordinate when Rocks tried to order him around in God Valley.
Even Sengoku described the crew as a bunch of pirates killing each other who couldn't work together.
As I said, Rocks was such a failure at trying to handle these people that Whitebeard rejected Oden because, from his experience with Rocks, he thought such breed of beast couldn't be handled.
Roger, on the other hand, could handle a crew with people like Rayleigh, Gaban or Oden, all of them top tiers who were also conquerors. Why did such monsters, not only his subordinates but enemies like Whitebeard, Shiki or Kaidou, admired and respected Roger yet treated Rocks like that?