No actually he said he can't remember the last time someone was fighting TOE TO TOE with him.
He was serious against Yamato and he was winning.
But he did not destroy her. She was able to stand her ground.
You are looking through the lense of a translated medium, if you aren't fighting someone "TOE TO TOE" that is equivalent to it not being
fully serious, by whatever metric you want to judge it.
He didn't use drunk form against her, and still wants her to be the Shogun of Wano, she used her DF to nullify some of his power but still reeled in pain afterwards.
You can say
partially serious, or whatever, but he wasn't going anywhere near all out against her.