Who told you Yakuza bosses are irrelevant?
They virtually get no panel presence or time outside of Hyogoro and one was about to kill himself from getting infected by the Ice Oni virus. They are there to simply characterize Wano a bit more than normal, but they are nowhere near a Scabbard in strength at this point, or else Oda would have shown it.
These characters won't fight the F6, it will most certainly be passed to the Strawhats/Supernova, as Franky appearing in front of Sasaki is pretty much implying this chapter.
Don't be surprised if Brook encounters Who's Who or Robin meets up with Sanji to handle Black Maria while he handles Queen, or if Chopper takes over the Page 1 or Ulti fight for Usopp/Nami (or they continue to fight them because its possible). These things are very possible, and way more likely than generic fodder fighting main villains.