Yeah, it felt iffy to me as well... I wanted to go back and reread some Wano stuff with Drake/Coby/Akainu & Co., but at the end of the day I couldn't be bothered as I'm not really interested in those clowns that Garp brought along.
We basically had a scene of Guernica (CP0 guy) confronting Drake and stating he better not think they don't know who he actually is, strongly suggesting that his identity as a marine should be a secret even for them. Which made sense if SWORD was a secret, extraofficial agency within the Marines that only its members knew about (and maybe was operating against the official interests of the organization); but now we have Koby speaking of it in front of Teach and Kuzan explaining they're
officially ressigned soldiers who kept their ranks but could be fired anytime —therefore CP0 should be expected to know about them, and Kizaru should be aware of Drake still being a marine.
So either Drake went on a secret mission as an otherwise known member of SWORD (a mission so secret that not even admirals and the top intelligence agency should know about, so for them Drake was a former SWORD member who chose piracy), or SWORD was supposed to be an extraofficial group but this was retconned.
SWORD is still a secret for the public (Teach didn't know it), but Drake's actual affiliation was treated as a secret even for the Government...