Yes, indeed. They basically admit that the only thing that saves Shanks from doing more than "a lot of damage" is Shanks being uninterested. And even then, Shanks to some degree is a wild card. We know no one fucked the WG over as much as Shanks did, stealing the Nika fruit out of the WG's hand and feeding it to a boy he inspired to become PK. Everything that happens now is a chain of events that started with Shanks. And even then, Shanks can walk in and out of Marijoa. Admiral stans think it's because he is somehow related to Garling, as if the Gorosei would give a rat's ass. They cast out Doflamingo despite Doflamingo was in the inner circle of the Tenryuubito and wanted to stay one, yet they suck off the great pirate that led to their demise because he is a Figarland?
Wano, a country coveted by the government for centuries, has never been conquered by the World Government. Currently, it's Kaido who instills fear in the World Government to such an extent that they have never been able to conquer Wano or challenge Kaido. Shanks can enter and leave Mariejois, speak on equal terms with the Gorosei, halt a war to which he invited both the Marines and Blackbeard to face him. He can stop Kaido, the monster, and Big Mom, with strong ties to Elbaf, almost managed to bring the power of the most powerful nation to herself - the giants, something the government couldn't achieve. And of course, there's the vast empire she has built.
Whitebeard, according to Sengoku, the Fleet Admiral of the Marines, had the potential to defeat the entire Marine force (soldiers) + admirals + Shichibukai + Pacifistas + Garp + Sengoku himself. Blackbeard, aiming for world domination, with a special lineage, possessing two Devil Fruits, something abnormal for anyone, and being hailed as the king of intellect, Luffy is the damn nika, the sun god, a god is a Yonko.
In my opinion, the Yonko ruling the world, that's the reality, they are the lions of the manga.