If garp, while fighting an actual admiral in his prime, is able to notice shiryu from far away and even outspeeds him, it shouldn't be far away from something zoro is already capable of. Remember, contrary to the misconception among parts of the fandom, zoro has extraordinary CoO feats already.
In the first "clash", even Aokiji got bodied.
What you describe is someone powerful and crafty. If a person only has stealth, intelligence and deadly sneak attacks in his repertoire, they're definitely more fitting into cp0. They'd not be fitting for a largely destructive, full on haki based death match. Feat-wise their battle would take 2 minutes. 1 for zoro adapting to his invisibility, 1 for OH him.
I wouldn't worry: he will look great.
An old garp should not be out of league of zoro's EoS enemy.
Now he is not at his peak and he is against a stronger foe.