We know that Sanji's Dad can casually one shot Vergo's superior clones whenever he wants to. Pretending there's an unlimited number of Vergo's fighting against Sanji's Dad, how many do you think it takes to kill his boredom? How many does it take to make Sanji's Dad tired?
IMO, Sanji's Dad could probably take down 200 Vergo's coming at him at once and maybe up to 2,000 if they fight him one at a time.