that's totally a headcanon, we didn't see the Gorosei speaking about anything like that.
what we saw is that Shanks was able to enter Mary Geoise without anyone noticing him or trying to stop him , then he speak to Gorosei and they listened to him.
they even sent their CP0 to negotiate with likes of Orochi to sell them weapons, imagine these guys needing weapons from Orochi, while they have their Holy knights. not only this, but Orochi was actually threatening them, because Kaido was backing him up.
Rocks D Xebec was trying to become the king of the world, he attacked an important place for WG, celestial dragons were there as well, he knows what should he do to become the world king, he literally wanted to set on the empty throne, someone like him must have the knowledge about the Holy knights, yet he challenged the World Government.
finally if these Knights were stronger than the Yonko, how will Dragon ever dream of beating them??