I am having issues configuring my container to point to my Kubernetes cluster with the command gcloud container clusters get-credentials. I get the following error.

ERROR: gcloud crashed (CannotConnectToMetadataServerException): <urlopen error [Errno -2] Name does not resolve>

If you would like to report this issue, please run the following command:
  gcloud feedback

To check gcloud for common problems, please run the following command:
  gcloud info --run-diagnostics

Enhanced logging:

CannotConnectToMetadataServerException: <urlopen error [Errno -2] Name does not resolve>
2018-04-10 18:00:42,625 ERROR    ___FILE_ONLY___ BEGIN CRASH STACKTRACE
Traceback (most recent call last):
  File "/google-cloud-sdk/lib/googlecloudsdk/gcloud_main.py", line 147, in main
    gcloud_cli.Execute()
  File "/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 818, in Execute
    self._HandleAllErrors(exc, command_path_string, specified_arg_names)
  File "/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 856, in _HandleAllErrors
    exceptions.HandleError(exc, command_path_string, self.__known_error_handler)
  File "/google-cloud-sdk/lib/googlecloudsdk/calliope/exceptions.py", line 526, in HandleError
    core_exceptions.reraise(exc)
  File "/google-cloud-sdk/lib/googlecloudsdk/core/exceptions.py", line 111, in reraise
    six.reraise(type(exc_value), exc_value, tb)
  File "/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 792, in Execute
    resources = calliope_command.Run(cli=self, args=args)
  File "/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 751, in Run
    self._parent_group.RunGroupFilter(tool_context, args)
  File "/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 692, in RunGroupFilter
    self._parent_group.RunGroupFilter(context, args)
  File "/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 693, in RunGroupFilter
    self._common_type().Filter(context, args)
  File "/google-cloud-sdk/lib/surface/container/__init__.py", line 71, in Filter
    context['api_adapter'] = api_adapter.NewAPIAdapter('v1')
  File "/google-cloud-sdk/lib/googlecloudsdk/api_lib/container/api_adapter.py", line 147, in NewAPIAdapter
    return NewV1APIAdapter()
  File "/google-cloud-sdk/lib/googlecloudsdk/api_lib/container/api_adapter.py", line 151, in NewV1APIAdapter
    return InitAPIAdapter('v1', V1Adapter)
  File "/google-cloud-sdk/lib/googlecloudsdk/api_lib/container/api_adapter.py", line 172, in InitAPIAdapter
    api_client = core_apis.GetClientInstance('container', api_version)
  File "/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/apis.py", line 297, in GetClientInstance
    api_name, api_version, no_http, _CheckResponse, enable_resource_quota)
  File "/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/apis_internal.py", line 153, in _GetClientInstance
    http_client = http.Http(enable_resource_quota=enable_resource_quota)
  File "/google-cloud-sdk/lib/googlecloudsdk/core/credentials/http.py", line 64, in Http
    creds = store.LoadIfEnabled()
  File "/google-cloud-sdk/lib/googlecloudsdk/core/credentials/store.py", line 281, in LoadIfEnabled
    return Load()
  File "/google-cloud-sdk/lib/googlecloudsdk/core/credentials/store.py", line 348, in Load
    cred = STATIC_CREDENTIAL_PROVIDERS.GetCredentials(account)
  File "/google-cloud-sdk/lib/googlecloudsdk/core/credentials/store.py", line 162, in GetCredentials
    cred = provider.GetCredentials(account)
  File "/google-cloud-sdk/lib/googlecloudsdk/core/credentials/store.py", line 214, in GetCredentials
    if account in c_gce.Metadata().Accounts():
  File "/google-cloud-sdk/lib/googlecloudsdk/core/credentials/gce.py", line 127, in Accounts
    gce_read.GOOGLE_GCE_METADATA_ACCOUNTS_URI + '/')
  File "/google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py", line 289, in DecoratedFunction
    exceptions.reraise(to_reraise[1], tb=to_reraise[2])
  File "/google-cloud-sdk/lib/googlecloudsdk/core/exceptions.py", line 111, in reraise
    six.reraise(type(exc_value), exc_value, tb)
  File "/google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py", line 159, in TryFunc
    return func(*args, **kwargs), None
  File "/google-cloud-sdk/lib/googlecloudsdk/core/credentials/gce.py", line 52, in _ReadNoProxyWithCleanFailures
    raise CannotConnectToMetadataServerException(e)
CannotConnectToMetadataServerException: <urlopen error [Errno -2] Name does not resolve>

To give some color, we kick off a build to CircleCI everytime we push code to github. However, we have a container we call internally belushi, that we use to run our entire infrastructure. This container has gcloud installed in it. CircleCI infrastructure is on AWS and when they spin up the belushi container we actually run gcloud get-credentials that point the belushi container to our project in google cloud, which has a kubernetes cluster configured and we run all of our functional CI testing in that cluster. So we need that belushi pod to configure into the ci project to move forward.

The weird thing is that the belushi:latest image always configures properly; however, when we are working on belushi we often branch and create a new image to run tests. So for example, I will create a branch in belushi and then have a new hash of 1234567, so we will spin up the belushi:1234567 image and try to run things, and the first thing we do is configure it to point to the ci project; however, we get that metadata resolve issue.

I feel like it is DNS related or maybe the metadata server isn't allow the new image of belushi to communicate with it right away. After I retry it a bunch of times it will eventually configure properly (without any code changes). So I wonder if the metadata server is rejecting it for some reason or it could be on AWS not resolving for some unknown reason.