We must remember, just because you can use a sword doesn't mean you have mastered or have full control of that sword.
Zoro had no problem using enma at the beginning of the war when he wasn't using haki.
Anybody that disagree better bring facts Oden had mastered enma at a young age.
PS: Zoro mastered enma in less than a mouth.