Them not controlling every single island there is doesnt change the fact that the world at large is under their command, and Spandan wasnt exaggerating, no one, not even Luffy, treated his speech that way
Robin stated that her enemy was the world i.e the WG
The empty throne is supposed to represent how the is no single king of the world in the WG
And even Whitebeard himself saw the WG as the ones that rule the world
Its not open to interpretation dude. Challenging the world=Challenging the WG, and that will be done by whoever finds the one piece and BB isnt that guy, he will fall during the race for the one piece