Was he supposed to fight Shanks, the red hair pirates, the straw hat pirates, the samurai, Law, Kidd, etc all at once?
Aramaki was literally going into a situation where'd he'd have to fight 20 Billion worth of pirates at once.
Of course he backed off.