I have an application that communicates with Google Cloud Custom Metrics from the cloud. The application is Deployed via GKE and running inside a pod. This application is running around 2 years and from time to time I getting a strange problem - the Google cloud credentials do not exist in the machine. As you may know, if you are running the application inside the cloud - the credentials already should be there. So, this is very strange.

This is the exception I get:

System.InvalidOperationException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
   at Google.Apis.Auth.OAuth2.DefaultCredentialProvider.CreateDefaultCredentialAsync()
   at Google.Api.Gax.Grpc.ChannelPool.CreateChannelCredentialsUncached()
   at Google.Api.Gax.TaskExtensions.WaitWithUnwrappedExceptions(Task task)
   at Google.Api.Gax.Grpc.ChannelPool.GetChannel(ServiceEndpoint endpoint, IEnumerable`1 channelOptions)

Killing the pod and start it again will make it work again.

It's only happened 2 times in the last two years.

For me, it seems like a bug on Google Cloud. Does anyone also notice this behavior? What I should check? Any solution?