I am running into the following quota limit:
jobs.Create: jobs.batch "..." is forbidden: exceeded quota: gke-resource-quotas, requested: count/jobs.batch=1, used: count/jobs.batch=50k, limited: count/jobs.batch=50kWhat is the recommended way to handle this limit? Is there a way to set job entries to expire after a specified amount of time?
Users may hit the same error when trying to create new pods, services, jobs, or ingress.
Cause:
The K8s resource quota controller occasionally fails to update the "used" values in the ResourceQuota objects.
To examine quota:
Use kubens command to know the active namespace. Use below command to examine the resource quota for active namespace. In yaml file, Used values shouldn't exceed hard resource limits.
kubectl get resourcequota gke-resource-quotas -o yaml
To examine resource quota for specific namespace use below command.
kubectl get resourcequota gke-resource-quotas --namespace=xxxxxx --output=yaml
Solution:
Deleting the gke-resources-quotas object is enough to lift the quota limit and proceed with object (pod, service, job) creation.
kubectl delete resourcequota gke-resource-quotas -n namespace-name
Note: The kubectl delete command needs to be applied in the right namespace, that is, the one where the pod/job/service has to be created.