Because never was the moment to say that to anyone more, but it's very clear than he thinks the same with the other nakamas. He would said the same to jimbei, brook or franky. Luffy always wanted to have a freak crew like mugis and sanjino is one of them. The phrase didn't have nothing to do with the sanjino's power like you want to say.