The thing is Sengoku is already retired, and he also already old, if warlords or more powerful characters fought the sword and their founder wont be able to protect them then its pointless
Drake resigned from Marine and become pirate (to did undercover mission) some years ago when Sengoku still a Fleet Admiral.
It implies that Sword isn't newly iniciated.
It is just recently revealed when we reach Wano Arc, but it could have operated from years ago
Also, becoming creator/founder doesn't necessarily means he is the leader now.