We know because narrative, and we know because we are trying to put some sense in Oda's nonsense.
However, if tomorrow a random character says "I won't hold back", we would say (rightfully) that character is serious. Even if not absolutely 100%, 90; 80 if we really want to stretch.
Which would still be world-class power level nonsense applied to Luffy surviving/evading Mihawk.
The point is Oda doesn't care: he needed pre-TS Luffy facing, surviving, evading, not going KO, etc., in the face of these dudes, and he didn't, by sheer plot-induced nonsense/strength, if you want.
For the same reason, Mihawk was stalled by Vista or Kuzan by Jaws. Crocodile is another example.
Usually, narrative is the real way to go, in particular regarding the interaction with pre-TS Luffy. Half of the character had haki, for example, but randomly didn't use it against him.