How do I remove the Google Calendar sync inside my CRM account? 

To cancel the one way sync so that your CRM account Calendar no longer reads from a Google Calendar you will need to remove the Google Calendar ICAL URL that was pasted into the Location Field inside each of the CRM account Calendars which read from a Google calendar.

Please see the screenshot example below which shows a Google Calendar inside your CRM account.  Just remove the ICAL address from the Location: field as shown below in red: 

Then click on Save (as shown above) to stop the one way sync reading from Google Calendar