Trolling aside, the reason why Whitebeard, Linlin and Kaido have to leave the area where Garp attacked is probably because they were surrounded not only by him, but by the traitors as well, reason why Whitebeard talked about "not dragging things out"
He fought Roger later in a 1 on 1, someone comparable if not stronger than Garp(younger, they clashed evenly and he might get better portrayal against Rocks), so we know it had nothing to do with fear