GKE reports GCE :Quota googleapi: Error 403: Insufficient Permission, insufficientPermissions
I've created a GKE cluster using
gcloud container clusters create mycluster \
--zone us-central1-f \
--num-nodes=2 \
--enable-autorepair \
--scopes default,datastore,bigquery,storage-rw \
--machine-type=n1-standard-2 \
--enable-autoscaling \
--min-nodes=1 \
--max-nodes=5
On this cluster, I've deployed Ingress controllers using the GCE
ingress class
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: myingcontroller
labels:
app: myapp
annotations:
kubernetes.io/ingress.class: "gce"
...
GKE successfully creates the load balancer and proper instance groups.
After about 4 days GKE is unable to modify/delete these same load balancers. Or create new ones. I get these events from Kubernetes
kubectl describe ingress myingcontroller
Name: myingcontroller
Namespace: default
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
18s 18s 1 loadbalancer-controller Normal ADD default/api-missions-api
18s 18s 2 loadbalancer-controller Warning GCE :Quota googleapi: Error 403: Insufficient Permission, insufficientPermissions
From what I can tell GKE load balancer controller is managing GKE resources using the default service account, this service account, however, seems to expire after a short number of days.
How can I make the service account's permissions persist? Is there a way to force the load balancer controller to use a different service account?
As a workaround, I can reset the k8s master password using
gcloud container clusters update mycluster --generate-password --zone us-central1-f
But this operation is expensive.