This is assuming the diamond only covers the outside of his body or skin instead of turning his entire body inside and out into diamond.
Think about Luffy or Katakuri and their DF. Their entire body is rubber and flower.
If that's the case attacks that cause internal damage won't do anything to him.
You've got a point, even though those examples do not match, as they are respectively a mythical Zoan and a special paramecia while Luffy's body is rubber in default state, whereas Jozu has to activate it.
If that fruits grants full invulnerability inside out tho, then it might come with a movement restriction. Iirc Jozu never moved those limbs, when they were covered in diamond.
Anyway I think Zoro and Luffy won't be stopped by that.
What's interesting is that if he really has that fruit, the diamond fruit seemingly makes a difference in the performance of a lunarian. Depending on what it is, if it's not just extra protection for his speed mode, things might get interesting in terms of Attack type and defense scaling.