It's even more absurd imo. Look, I'm not that much of a Vegapunk fan but the anti Vegapunk arguments are flawed af. This guy is inspired by Einstein and is consistently portrayed as the smartest character in this verse. Oda clearly didn't intent to make him a fraud at all. Now we find out that he actually has a DF to lift up the limitations of his memory storage - Vegapunk is so smart, no "human brain" has the capacity to store all the information which VP is capable to gain. Imagine how smart you have to be that your intelligence surpasses your actual brain capacity...
Also, it's a difference if you memorize everything or if you can effectively use this amount of information. Analytical and intellectual skills are a must for a scientist like VP. You won't go this far if you just memorize everything in Computer Science for example or if you're a system or software engineer and you are about to develop something. What matters is the fact how good of a problem solver you are. Thus, VP also has to know how to use his knowledge.