I am trying to use the Application Default Credentials on Container Engine, for the python IAM API. But I am getting the following error pointing to insufficient authentication scopes. My project has the IAM API enabled, and the code works locally. So, I am not sure what I am missing.

My Error Message:

22:26:16.000
ERROR:root:<HttpError 403 when requesting https://iam.googleapis.com/v1/projects/henry-dev/serviceAccounts/test123@henry-dev.iam.gserviceaccount.com/keys?alt=json returned "Request had insufficient authentication scopes.">
{
 metadata: {…}   
 textPayload: "ERROR:root:<HttpError 403 when requesting https://iam.googleapis.com/v1/projects/henry-dev/serviceAccounts/test123@henry-dev.iam.gserviceaccount.com/keys?alt=json returned "Request had insufficient authentication scopes.">
"   
 insertId: "116wpgtg3n4zndx"   
 log: "simplekubeserver"   
}

22:26:16.000
HttpError: <HttpError 403 when requesting https://iam.googleapis.com/v1/projects/henry-dev/serviceAccounts/test123@henry-dev.iam.gserviceaccount.com/keys?alt=json returned "Request had insufficient authentication scopes.">
{
 metadata: {…}   
 textPayload: "HttpError: <HttpError 403 when requesting https://iam.googleapis.com/v1/projects/henry-dev/serviceAccounts/test123@henry-dev.iam.gserviceaccount.com/keys?alt=json returned "Request had insufficient authentication scopes.">
"   
 insertId: "116wpgtg3n4znej"   
 log: "simplekubeserver"   
}

My Code, which works locally but not on GKE:

from oauth2client.client import GoogleCredentials

def _iam_service():
  credentials = GoogleCredentials.get_application_default()
  return discovery.build(serviceName='iam', 
                         version='v1',
                         credentials=credentials)

def list_keys(project_id, service_account_id):
  full_name = 'projects/{0}/serviceAccounts/{1}'.format(project_id, service_account_id)
  keys = _iam_service().projects().serviceAccounts().keys()
  request = keys.list(name=full_name)
  return request.execute()

One thing I did to troubleshoot is to get which service account is being used.

print credentials.service_account_email

Locally, this shows the correct service account I am using. Whereas on GKE, I get None, but expected something like 123456789-compute@developer.gserviceaccount.com

In the source code, I see:

_get_application_default_credential_GCE()          
_get_application_default_credential_GAE()

But nothing explicit for GKE. So, I am assuming the one for GCE is used.

This doc mentions that this should be working on Container Engine.

Application Default Credentials are best suited for cases
when the call needs to have the same identity and authorization level
for the application independent of the user. This is the recommended
approach to authorize calls to Google Cloud Platform APIs, particularly
when you're building an application that is deployed to Google App
Engine, **Google Container Engine**, or Google Compute Engine virtual
machines.