Members will not be added to your contacts until they complete the registration form that is sent to them when they first text you. They need to click the link that was sent out with your welcome message so they can opt-in to your Community and be added to your contacts list. No messages will come through to you until they complete this form. Similarly, no messages you send will go through to them.
Additionally, if a member has opted out (by texting STOP to the number), you will no longer see them as a contact in Community.