This may be because you have published your public ICAL public address and not your private ICAL address for your Google Calendar inside your CRM account.
For the calendar read from Google into your CRM account calendar to work you must use your private ICAL address as shown below:
Please click here to find the instructions that you need to correctly set this up.