Well we saw how he acted when pushed to the brink against Oden. And how he tactfully shat himself against the Scabbards when he thought they were hot shit.
We didn't? All we saw was Kaido hitting him lmao.
We don't know his thought proccess nor actually anything.
Again, the manga debunked what you said about the scabbards.
Not once was he scared of them, he watched them, used some breath attacks and took theirs.
When he saw that they couldn't kill him he decided to end it, but not before telling them that he wouldn't have minded dying to them and that death completes a man.
Yet for you, you twist what actually happened into something that suits you and then decide that he was actually scared of them(when the manga shows you and tells you he wasn't) and decide that he is a man scared of death.