GKE Default Service Account expires

8/23/2017

GKE reports GCE :Quota googleapi: Error 403: Insufficient Permission, insufficientPermissions

I've created a GKE cluster using

gcloud container clusters create mycluster \
        --zone us-central1-f \
        --num-nodes=2 \
        --enable-autorepair  \
        --scopes default,datastore,bigquery,storage-rw \
        --machine-type=n1-standard-2  \
        --enable-autoscaling \
        --min-nodes=1 \
        --max-nodes=5

On this cluster, I've deployed Ingress controllers using the GCE ingress class

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myingcontroller
  labels:
    app: myapp
  annotations:
    kubernetes.io/ingress.class: "gce"
...

GKE successfully creates the load balancer and proper instance groups.

After about 4 days GKE is unable to modify/delete these same load balancers. Or create new ones. I get these events from Kubernetes

kubectl describe ingress myingcontroller
Name:            myingcontroller
Namespace:        default
Events:
  FirstSeen    LastSeen    Count    From            SubObjectPath    Type        Reason        Message
  ---------    --------    -----    ----            -------------    --------    ------        -------
  18s        18s        1    loadbalancer-controller            Normal        ADD        default/api-missions-api
  18s        18s        2    loadbalancer-controller            Warning        GCE :Quota    googleapi: Error 403: Insufficient Permission, insufficientPermissions

From what I can tell GKE load balancer controller is managing GKE resources using the default service account, this service account, however, seems to expire after a short number of days.

How can I make the service account's permissions persist? Is there a way to force the load balancer controller to use a different service account?

As a workaround, I can reset the k8s master password using

gcloud container clusters update mycluster --generate-password --zone us-central1-f

But this operation is expensive.

-- Curtis Allen
google-kubernetes-engine
kubernetes

0 Answers