For the people that missed, Zoro only said King's defense drops only a little when the flame is off, so it's mostly from his race and of course, his zoan fruit plays a role too. Besides, Lunarian toughness isn't just about durability. VP was conducting endurance tests on King, meaning the dominant attribute about them is endurance, not just durability.