An army of Giants would still get low diff'd by a strong high tier. But compare them against the other armies? They wash them.
That's exactly why Kaido's going for the SMILE devil fruit users, Big Mom wanted Giants in her army and then Germa's solidiers. They are all elite fodder but still fodder.
Zoro's statement is like comparing King's strength to Giants, but it should be understood as Lunarians possessing physical strength comparable to Giants. King is a YFM, ofcourse he is much stronger than an average Lunarian or an average Giant.
Whitebeard had Ancient Giants, Fishmen and Minks on his ship and yet he hyped Lunarians as "gods". They might be one of the strongest races in the verse probably, on an average.