Yes, he could have, but still, since Gaban was talimg about damaging them, I don't see another intepretation. Not to mention Harald was Xebec's peer, so him being able to do the same is kind of sure.
He can be wrong, but as anybody. Also, Imu wasn't the only one to hype Harald to the high heavens:
Imu was wrong... but because someone has eaten the legendary fruit and got Ragnir.