Because Shiryu isn't that strong. There's a reason why Shiryu was in prison. Because Ivankov was straight up wrong. Magellan>Shiryu. Regardless. We know Shanks was hunting Blackbeard down despite the fact that Kuzan is a titanic captain. Because the RHP are still stronger than the BBP. Kuzan included.

Egghead : 4 ppl
Amazon Lily : 3 ppl
He needs to leave people behind to protect his main island at all time