Exactly man.
People who really don't believe him in facing Shanks are just terrified of Shanks facing him and losing inevitably, also falling victim to Gear 5 and its trolling too.
Shanks and Luffy not fighting would be like Zoro and Mihawk not fighting. Even if they Davy Back Fight, they still have to fight for the finale of it, like Luffy and Foxy did it too.
And there's no way in hell Luffy should take what happened to Bartolomeo lightly, ideally with Kidd too. He needs to see the complete Shanks, not just the idealised hero, the do gooder idol he's seen. He needs to see the nastier sides to Shanks. Same for Usopp with his own father too. They need a serious wake up call and like with Natsu and Gildarts, nobody is more appropriate in the best ways possible to deliver that wake up call, loud, rude and clear.
Luffy not fighting Shanks would be fucking terrible, ireedeemable and truly cowardly writing from Oda.
Some are trying to claim he won't fight him because he's holding only 3 fingers up which I fucking hope isn't "foreshadowing" but just a mistake by Oda or just a gag by him with how dumb Luffy is. But given what Oda is like, I expect the worst unfortunately.