whilst 'hardening' the accounts - namely removing or toning down accounts with editor permissions on the projects I removed editor from what appears to be the kubernetes account that container engine uses on the back end of gcloud commands.

Once you remove the last role from an account it vanishes - hard lesson to learn! Removed editor serviceAccount:386242358897@cloudservices.gserviceaccount.com

• It meant I initially couldn't deploy because it couldn't access container registry.
• So I deleted the cluster and recreated expecting the account to get recreated. That failed due to insufficient permissions.
• so I manually removed the compute instances (it wouldn't have permissions to recreate them), then templates and then the cluster.

• As the UI now thinks you have no clusters it looks like you are back to the beginning. So I ran my scripts and they failed.

  ERROR: (gcloud.container.clusters.create) Opetion [https://container.googleapis.com/v1/projects/xxxx/zones/europe-west2-b/operations/operation-xxxx' startTime: u'2017-10-17T17:59:41.515667863Z' status: StatusValueValuesEnum(DONE, 3) statusMessage: u'Deploy error: "Not all instances running in IGM. Expect 1. Current actions &{Abandoning:0 Creating:0 CreatingWithoutRetries:0 Deleting:0 None:0 Recreating:1 Refreshing:0 Restarting:0 Verifying:0 ForceSendFields:[] NullFields:[]}. Errors [https://www.googleapis.com/compute/beta/projects/xxxx/zones/europe-west2-b/instances/gke-xxxx-default-pool-xxxx:PERMISSIONS_ERROR]".' targetLink: u'https://container.googleapis.com/v1/projects/xxxx/zones/europe-west2-b/clusters/xxxx' zone: u'europe-west2-b'>] finished with error: Deploy error: "Not all instances running in IGM. Expect 1. Current actions &{Abandoning:0 Creating:0 CreatingWithoutRetries:0 Deleting:0 None:0 Recreating:1 Refreshing:0 Restarting:0 Verifying:0 ForceSendFields:[] NullFields:[]}. Errors [https://www.googleapis.com/compute/beta/projects/xxxx/zones/europe-west2-b/instances/xxxx:PERMISSIONS_ERROR]". Updated property [container/cluster].

when I try to create through UI I get this

Permission denied (HTTP 403): Google Compute Engine: Required 'compute.zones.get' permission for 'projects/xxxx/zones/us-central1-a'

Have done a number on it! My problem is that I don't see a way of giving permissions back to whatever account it is trying to use (as I cannot see that account if it exists) nor can I see how to attach a new service account with permissions that are needed to whatever is doing the work under the hood.

UPDATE:

So ...

I recreated the account at the organisation level. Gave it service account role there because you cannot modify the domain of the accounts at project level.

I have then modified that at the project level to have editor permissions.

This means i can deploy a cluster but ... still cannot create load balancer - insufficient permissions

Error creating load balancer (will retry): Error getting LB for service default/bot: googleapi: Error 403: Required
 'compute.forwardingRules.get' permission for 'projects/xxxx/regions/europe-west2/forwardingRules/xxxx', forbidden

the user having the problem this time is: service-xxx@container-engine-robot.iam.gserviceaccount.com