Nusjuro is very obviously a Zoro opponent, especially after last chapter.
The thing with Topman (and most of the other Gorosei) is that he hasn’t showcased that he’s actually an expert combatant.
He can have tremendously powerful haki and supreme durability, but can he actually use his strengths effectively? He comes across as nothing more than a wild animal so far.