Garp-san has a very tarnished history; his title of hero was falsely attributed to him. He fought to protect the scoundrels who committed all kinds of atrocities on that island. As for Roger, there's some justification because Xebec was a pirate like him, and a pirate who had more notoriety, fame, and power than Roger at the time. By defeating him, Roger could have boosted his career as a pirate. There are various reasons why Kaido might not have placed Garp in his top 5 capable of fighting against him. Although I have Garp above all the admirals in terms of individual power, even after these blemishes on his record, being called a hero for something that was falsely spread, a fake news that he defeated ALL the ROCKS pirates alone, is absurd, and that's why Garp doesn't like to talk about it.