In general, being an adaptation, the anime is its own thing.
Should be analyzed separated.
Now, there are cases of "multi-media" connections.
Avatar the last air bender extends its canon universe throughout comics, for example. Despite being different medias, they serve as complements for each other.
If the anime staff intentionally fills gaps from the manga, and it DOESN'T CONTRADICT THE MANGA IN ANY WAY, just adding to it, and is also explicitly "green light'ed" by the original author, it is actually OK to use it as part of the original story.
Oden's flashback might be the case, if that tweet someone posted saying that the anime staff was going to get the personal "ok do it" signal from Oda and be strictly faithful to the manga while adding scenes, then its pretty fair to consider it canon.
But if Oda says: "no, its not", or it directly contradicts events from the manga, then its not.