Yes, but the Gorosei are clearly of another level of intelligence. You could say that not only just looking and hearing them, but even from the fact that they don't go around with that stupid fishbowl on their heads outside Marijoa.
I don't know if He was afraid for his life, but certainly he doesn't want problems, because being accused of the murder of a CD is a problem. If push comes to shove, he will fight, but obviously he can't do that right now.