If anything applying Coa/CoC to a sword requires more skills than applying it to the body (be it fists or legs), because you've to imprint your Haki to an external tool.
So if they can imprint their Haki on their swords they can do the same with their fists as it requires less skills.
The opposite doesn't and wouldn't make sense logically speaking.
Fucking Hyogoro without a sword wasn't helpless against opponents of his level and here some geniuses suggest that Mihawk and Zoro would be done without swords because...because "I said so".
As if their physical strenght, endurance, durability, Haki, agility, speed, reflexes etc. would be gone without their swords