Yes, sanji kindness did play big role in swaying Luffy but you have to see the narration as well
Shs arrived at Baratie with Johnny and yosaku talking about how good chefs and food is there. Luffy was looking for a cook.
Then you we have Sanji serving food to Gin - his act of kindness
And, we have Gin making a remark - "he has never tasted such delicious food "
And, then luffy decided he will have sanji as cook.
But yeah - luffy invited sanji without tasting his food so I guess you have a point