This is a misconception.
Of course we have speculated about a lot of the points Vegapunk brought up (Joyboy vs Imu war of ideals, there being continents 1000 years ago, current islands only being the tip of those continents etc.) but Oda has never clarified it in the manga.
You have to think of it from the perspective of an average fan who doesn't spend most of his free-time on OP discussion forums.