Luffy is utterly superior to everyone in his crew, its obvious they accept him.
Rocks on the other hand was an alistar group full of yonkos and conquerors, they might not accept him, but still follow the guy.
For all WB talk, he was still there following rocks, wasn't him?