I can't create new pods in my default namespace in GKE. I checked the error on deployments and replicasets using kubectl -n default get replicaset xxxx
--> The error is:
Warning FailedCreate 6m54s replicaset-controller Error creating: pods "dummy-deployment-54b6f555b7-ms5sb" is forbidden: exceeded quota: gke-resource-quotas, requested: pods=1, used: pods=5k, limited: pods=5k
So I ran this command kubectl get resourcequota gke-resource-quotas -o yaml -n default
to look at what's going on with gke-resource-quotas
I get this result:
apiVersion: v1
kind: ResourceQuota
metadata:
creationTimestamp: "2020-02-04T13:06:27Z"
name: gke-resource-quotas
namespace: default
resourceVersion: "109072XXXX"
selfLink: /api/v1/namespaces/default/resourcequotas/gke-resource-quotas
uid: 2727d85b-474f-11ea-a2f2-XXXXXXXXXX
spec:
hard:
count/ingresses.extensions: 5k
count/jobs.batch: 10k
pods: 5k
services: "1500"
status:
hard:
count/ingresses.extensions: 5k
count/jobs.batch: 10k
pods: 5k
services: "1500"
used:
count/ingresses.extensions: "7"
count/jobs.batch: "4540"
pods: 5k
services: "20"
But I checked using kubectl -n default get pods | wc -l
I do not have 5k pods running.
This is preventing me from creating new pods in my default namespace in GKE.
This seems like an error on part of GKE/K8s. Does anyone know how I can resolve this? Thanks!
It totally looks like a GKE issue with their pre-built quotas. You might have at one point hit the 5k limit perhaps not being updated for terminated pods and it thinks it has reached the limit.
Someone else found a workaround so I would just try that for now:
kubectl delete resourcequota gke-resource-quotas -n default
If you are running at that scale and this is an important recurring issue I would strongly recommend hitting GKE/GCP support. Or seek alternatives like running outside of GKE.
✌️