I mean if the Holy Knights are the opponents of every single commander level ish character in the final war i can see that
if they fight the revs, Marco, Hancock, supernova, Katakuri, etc.
the Nerona family is likely a regular family outside of Imu. Ivankov mentioned there was an Imu that was just another 20 king, nothing special about him.
Figarland are just built different and special
besides Garling is going to be a Gorosei for most of the story
in Oda's mind, Shamrock may be the only Figarland Holy Knight