Give everyone equal haki all maxed out and equal base physical strength/speed
The swordsman isn't beating the person with the magma fruit
99.999% of the time if haki and physical strength is the equal the person with the devil fruit is gonna win. Especially when that fruit is a logia