That is a condition that you have placed, in OP the size thing doesnt matter if the author doesnt want it which is clearly the case here. Rocks aint that much bigger than Shanks and Gaban yet Harald was fighting him on panel too.
Nope, Oda has not drawn a Giant ever landing a flush blow on another nimble top tier for a reason. Usually what happens are clashes or moments where they overpower opponents. Harald just bullied Shanks and Gaban, especially considering he is unkillable to them so he kept regenerating, so their blows never counted, eventually he was going to wear them down, which he did.
They're not having a Lucci vs Luffy, or Kaidou vs Luffy type of back and forth.
Given the size of Harald's weapon the scar on Gaban cannot be a result of a direct attack landing clean to begin with. You realize Harald's weapon is many times bigger than Gaban's entire body, he wouldn't just be scarred on his face his whole body would be scarred or in tatters if a blow got to him.