I am trying to enabled Workload Identity on my cluster but I'm getting the following error when testing it from a pod:
root@workload-identity-test:/# gcloud auth list
ERROR: gcloud crashed (MetadataServerException): HTTP Error 500: Internal Server Error
If you would like to report this issue, please run the following command:
gcloud feedback
To check gcloud for common problems, please run the following command:
gcloud info --run-diagnosticsI've enabled workload-metadata-from-node:
gcloud beta container node-pools update NAME --cluster NAME --workload-metadata-from-node=GKE_METADATA_SERVER
Updating node pool main...done.Even if I add --log-http to the command it doesn't seem to show the request that's returning 500 (nor crucially, its response).
Any ideas? (I get the same error on a new cluster)
I got same problem, but I could resolved it by just enable "IAM Service Account Credentials API" on the console. https://console.developers.google.com/apis/api/iamcredentials.googleapis.com/overview
Update I have been able to get this working with workloadIdentityUser since. I suggest following the delete-recreate tips outlined in John's Answer if you still run into issues.
Based on errors logged by the gke-metadata-xxxx pod on the node where the test was running, I needed to use the roles/iam.serviceAccountTokenCreator instead of the roles/iam.workloadIdentityUser suggested in the tutorial linked above.
This allows me to use the feature as expected, however since the role is not documented I'm not sure exactly what additional permissions I might have granted by making this switch.
I was having an issue today about this. Everything was working fine using the roles (roles/iam.serviceAccountTokenCreator and roles/roles/iam.workloadIdentityUser) and my Google Cloud Credentials API is also enabled.
If ever you encountered this problem again even after ensuring you have all the requirements above, delete the Kubernetes service account and create it again. You also have to restart the Pods that are using the service account so they can get the fresh service account token.
Finally, everything will start working again.
roles/iam.serviceAccountTokenCreator impersonates service accounts (create OAuth2 access tokens, sign blobs or JWTs, etc) and roles/roles/iam.workloadIdentityUser impersonates service accounts from GKE Workloads. Both roles are documented here and you can see the permissions each role has.