1. He hasn´t seen Luffy up in person up until this point, but he didn´t have any interest.
2. Oda didn´t have Imu react to Luffy´s awakening in Wano like how he had Imu react to Loki´s killing of Harald.
3. Loki specifically has one of the only 2 fruits that allow you to fight Imu, Luffy doesn´t but tbf, it´ll most likely be added to the list at some point, but the buildup is there...
It´s pretty consistently being portrayed that Loki has a bigger connection to Imu at the moment, though obviously that will change by the end of the manga.