I think its sort of similar to how Doffy was treating Sanji's kicks in some respects.
Like yes, they acknowledge Sanji's kicks posssess some strength, but they aren't terribly effective.
I think that Sanji getting frustrated that he wasn't doing any major damage to Queen was sort of confirmed in the chapter where he talked about how tough Queen's hide was.
Queen at a later point even says that Sanji isn't putting up much of a fight.
Anyway, I think that Sanji was doing roughly as expected against a top emperor commander like Queen. Just look at Luffy's fights with Cracker and Katakuri, and we can see its following a similar trajectory of losing quite badly or getting nowhere to begin with before eventually powering up/improving/ changing fighting style, allowing them to come out on top.
Like, even Zoro, who has some ridiculous feats, is currently struggling with King.
Not entirely related, but I think its a good discussion. I think that Sanji struggling with Queen was pretty expected.