Eh I don't think fists physical size matters lol..its the size of your Haki output.
Zoro created an Island sized barrier to fully stop Hakai from proceeding for a little bit with tiny ass swords.
Likes of Mihawk Gol D Roger Shanks Rayleigh should be able to do a lot more.
And it doesn't just apply to swordsmen. Prime Garp has nothing but his fists and I would still bet prime Garp's strongest punch should be able to overpower Blazing Bagua Dragon and defeat Kaido.
For luffy his attack always depend on his DF and we talking pre haki with gear 2 and 3 .
Even with something like gear 4 when he beat don it was with haki and DF .
Haki is a factor but luffy DF will also be a huge factor in his speed , AP , AoE , defence etc etc .
With is why he get Gear 5 , Oda will never change how important his DF is to him in every way even with haki .