They don't have to and meeting the Tobi Roppo next chapter is a guarantee. The Numbers are shrouded in mystery. Oda kept them in silhouettes even in Oden's flashback. Are they truly giants or are those their Zoan forms? Their collective title suggest being the missing link after the Disasters (King, Queen, Jack, and then the Numbers). I know others would argue the Tobi Roppo, but I see more evidence pointing to them being Kaido's subordinate captains (hence why Hawkins and Drake are placed there).