Maybe there will be something really similar to a big final showdown at this point.
Buggy is a joke character but on one hand his "I will be pirate king" joke has serious chances to actually happen now and he is also backed by Mihawk (and Croco) one way or another (maybe he will also get some more ex shikis or strong allies eventually) so even if he is the intruder in this group he actually will most likely stay until the end and maybe even become PK at the end.
Shanks and BB are 2 sides of the same coin. Shanks is the Roger of the mid gen, the inspiration of the main character, in a way a sort of guardian of the One piece world who keeps peace. A sort of bright selfless "god". While BB is his perfect opposite, a bastard selfish dark "god". Almost a sun and a moon if Luffy wasn't already the sun. So imo they are both staying until the end when BB probably kills Shanks. Or that happens shortly before Luffy and BB clash.
The weaker contenders, at the very least going from portrayals (I mean importance for the story, in relation to Luffy), have already been dispatched (and I'm not even talking about Law and Kidd).