People need to realize that Luffy is "weak" to swords, in comparison to being "immune" to blunt damage. You can damage him more easily with Swords, than you can with blunt force.
He does not, however, receive more damage from a Sword attack, than your average person. So it's not a true weakness in the same vein like fire is weak to water, ice is weak to fire, or robots are weak to lightning, etc...
Durability differences aside, Rubber is Enels weakness, but Enel doesn't take more damage from Luffy's punches than Arlong does.
Logia intangibility aside, Crocodile is weak to water, but a water coated punch from Luffy doesn't deal more damage to him than if he landed a normal punch on Crocodiles body.
It's the same principle for Luffy being "weak" to swords.