Kaido already defeated Kidd and Killer when he said that Doffy was weak, hence he already saw the strength of two top supernovas. Even after Luffy defeated Katakuri, Kaido still negged him.
Imagine that even a fraud thinks lowly of you whilst he's consistently praising the samurai or other characters... I have a hard time defending Doffy guys.