The call for adventure is a part of the "heroes journey" theory. Its heavely used in modern fiction because of the influence of this theory and Oda doesn't do the exception and uses it too.
Here is how the refusal of the call for adventure works:
First, there is a call for adventure. Usually its a conflict or a task that the protagonist must resolve, other time its someone asking the protagonist to help or come with them. There are various types of call it would need a huge analysis. Anyway..
Secondly there is the refusal. The refusal is the protagonist being anxious in front of the call and or refusing it in one way or another. It can also take a lot of face, sometimes its the protagonist postpoining and being anxious in front of the responsibility they are facing, sometime the protagonist doesn't accept what there they are or the reality they are in, sometime its the protagonist straight up refusing the call.
For example Usopp in the manga doesn't really refuse the call for adventure (maybe Oda had already in mind the even of Water seven for Usopp) at first. But in the live action, Usopp does:
What is important to understand is that a refusal is temporary. Its usually takes a very short time.
THat's why each time in One Piece, the refusal is temporary, sometimes its very short (a few lines of dialogues) Zoro / Sanji /Franky and sometimes its takes longer and can be the center of a huge conflict (Nami / Robin)
Each mugiwara refuse the call, soon or later in the adventure. They refused to follow Luffy for a specific reason of their own but came back around and joined him anyway later.
The refusal of the call adds conflict, which is important in a story. That's why if you want to look for a nakama, look for the refusal of the call of adventure.