Yeah and talked about how Sanji is '#2' in this arc even King went after him. They go after those who are already injured, doesn't mean they don't see everyone else also a threat too especially ones on Strawhats with Zoro and Sanji are the top guys.
He said he been fighting them awhile, not only that moment. It doesn't make sense if it was that scene on him saying that. Getting push back doesn't mean fight is done when he got back up and hurt Queen and fight King too.