And yeah, if he doesnt care about Garp, it means he wont care about Luffys lineage, so a fight between them would probably not make sense. We only have one character in the story who cares about Luffys lineage
He also won't let Luffy get away of his crimes of piracy, where's Imu only cares about the crimes of the Davys, unless they invade his castle, I guess.