This is why it's important to pay attention to context. Law actually offered that Doflamingo could get Ceasar back and maintain his deal with Kaido if he gave up his warlord title.
As Law points out, he would have the attention of the navy admirals on him, they would chase him down. And Doffy doesn't like that either.
But we know from Doffy's interactions with Kuzan and Fujitora, he doesn't fear the admirals personally. He nearly kicked Fujitora and initially tried to kill Smoker in front of Kuzan. Put 2 and 2 together. It's not that he's worried about himself, he simply knows any top tier can kill plenty of his crewmates, who he considers like his "family"