People in East Blue didn't even know Shanks was a Yonko, 6 years after he became one. 99.9999% of the World probably don't even know about the Gorosei, let alone thier names.
Those who participate in Levely do
It's not about how many know or read about Gorosei
It's about the few who did, why didn't They question Gorosei's Age
Ivankov for example, He read about OG Founders, and as One of Revo Leaders, He obviously read about their Enemies
Why didn't He say that Gorosei lived for too long, when They were questioning Imu's Age
I'm not saying Gorosei can't be that Old, actually according to Sandman, Gorosei seem to be Confirmed as +200 Years Old
But i'm just saying that They possibly didn't live this long taking same Identity
It's possible They keep changing their Bodies & Names whenever needed to fool all the Kingdoms