This is because the sync is 1 way (from your CRM account to your iPhone or Android mobile).  


If you remove the sync tag you set up from your CRM contacts when your phone next 'syncs' it will only sync the contacts that have the sync tagged applied. 


Any contacts who were synced previously and who now do not have the sync tagged applied to their contact record will  now not be displayed as a contact on your iPhone/Android mobile.