GKE resourcequota INCORRECTLY shows pod limit reached and prevents new pod creation

8/21/2020

I can't create new pods in my default namespace in GKE. I checked the error on deployments and replicasets using kubectl -n default get replicaset xxxx --> The error is:

  Warning  FailedCreate  6m54s replicaset-controller  Error creating: pods "dummy-deployment-54b6f555b7-ms5sb" is forbidden: exceeded quota: gke-resource-quotas, requested: pods=1, used: pods=5k, limited: pods=5k

So I ran this command kubectl get resourcequota gke-resource-quotas -o yaml -n default to look at what's going on with gke-resource-quotas

I get this result:

apiVersion: v1
kind: ResourceQuota
metadata:
  creationTimestamp: "2020-02-04T13:06:27Z"
  name: gke-resource-quotas
  namespace: default
  resourceVersion: "109072XXXX"
  selfLink: /api/v1/namespaces/default/resourcequotas/gke-resource-quotas
  uid: 2727d85b-474f-11ea-a2f2-XXXXXXXXXX
spec:
  hard:
    count/ingresses.extensions: 5k
    count/jobs.batch: 10k
    pods: 5k
    services: "1500"
status:
  hard:
    count/ingresses.extensions: 5k
    count/jobs.batch: 10k
    pods: 5k
    services: "1500"
  used:
    count/ingresses.extensions: "7"
    count/jobs.batch: "4540"
    pods: 5k
    services: "20"

But I checked using kubectl -n default get pods | wc -l I do not have 5k pods running.

This is preventing me from creating new pods in my default namespace in GKE.

This seems like an error on part of GKE/K8s. Does anyone know how I can resolve this? Thanks!

-- crossvalidator
google-kubernetes-engine
kubernetes

1 Answer

8/21/2020

It totally looks like a GKE issue with their pre-built quotas. You might have at one point hit the 5k limit perhaps not being updated for terminated pods and it thinks it has reached the limit.

Someone else found a workaround so I would just try that for now:

kubectl delete resourcequota gke-resource-quotas -n default

If you are running at that scale and this is an important recurring issue I would strongly recommend hitting GKE/GCP support. Or seek alternatives like running outside of GKE.

✌️

-- Rico
Source: StackOverflow