It's no dumber than Zoro not recognizing the Seraphim as clones of King by their appearances but strictly by their durability after a long clash lol. Hell, he recognized Mihawk instantly and Daz Bones' fruit the second he saw it in use.
Zoro's always had these dumb gags and this is all this is, don't take it that seriously.