Snakeman Luffy is the only instance we actively have of Luffy trying to hit Kizaru, and if we interpret the panel correctly, he was apparently hitting Kizaru's leg, etc... After that, Kizaru started to keep his distance, and Luffy never saw Kizaru as a target anymore, he didn't direct attacks towards Kizaru, he stopped trying to hit him. Because if we were really talking about trying to hit someone, Luffy treating Kizaru like a toy is him doing that, he's toying with Kizaru...
But putting that aside, we don't have any other attacks from Luffy towards Kizaru, which shows how casual he is about it. It's Kizaru who is actively trying to hit and keep Luffy busy, and failing every time