There is a Sync flag that you must enable in the Contact in the CRM. Each user must enable this flag to get the contact to sync to their Outlook installation.