Yes, he called off because of Anbu because Kakashi, Gai, Kurenai, and Asuma were not really a deterrent. This is because Kisame wouldn't believe any of those guys were stronger then Itachi. This isn't true for Jiraiya, who Kisame does view as above Itachi, despite Kisame knowing about all of Itachi's accomplishments and hype. We can't just say Itachi is telling a white lie when his literal partner backs up his claim.
But Itachi is 100% telling a lie and is a known liar lol. That was a huge part of his development in the Kabuto fight, admitting this several times.
Itachi also states that if they attacked Jiraiya together it would be meaningless, which is another massive lie. You throw any Akatsuki pair at Jiraiya and he's losing.
Itachi needed a convenient excuse to leave and Jiraiya was that excuse. Kisame does what Itachi says, Itachi tells him not to be flashy and Kisame isn't flashy, Itachi says to run, they run. That's their dynamic.
It doesn't really matter how close Jiraiya was to figuring out his secret, Pain is only really worried about Jiraiya figuring it out because he doesn't think anyone else in the leaf can do anything to him if they figure it out. He has a god complex and believes himself to be close to invincible.
No other shinobi was in a position like Jiraiya to figure it out or were as accomplished? Kakashi didn't have a clue and posed no threat to Pain in that manner, as far as Nagato was concerned.
Jiraiya knowing the secret is bad because he was in an environment where he could (and did) slip away and outwit Pain once with the Frog Genjutsu. Imagine if Jiraiya knew about the main body and used what spare time he had to slip away and look for Nagato? Catastrophic.
Kishi is only having these Akatsuki guys hype up Jiraiya as a threat because that's the narrative, simple as. Jiraiya exists in the story to be someone strong enough to protect Naruto from the Akatsuki. They don't ever hype up any other leaf ninjas really, not even someone super strong like Kakashi.
Kakashi's seen as a scrub for most of the series but still gets hyped. Deidara recognizes him as troublesome, Pain is honored to meet someone so prolific, etc. But Kakashi isn't a sannin lol, the Sannin name carries weight to it.