Resource Quota applied before LimitRanger in Kubernetes for Pod without specified limits

6/4/2020

While using Kubernetes v1.16.8 both the ResourceQuota and LimitRanger are enabled by default and I did not have to add them in my admission plugin in kube-apiserver. In my case, I use the following LimitRanger

apiVersion: v1
items:
- apiVersion: v1
  kind: LimitRange
  metadata:
    name: mem-limit-range
    namespace: test
  spec:
    limits:
    - default:
        memory: 512Mi
      defaultRequest:
        memory: 256Mi
      type: Container

and it adds the default limit for memory usage in a new Pod without specified limits, as expected.
The Pod's definition is as simple as possible:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: test-pod-ctr
    image: redis

When I get the created pod described it has acquired the value for limit from the LimitRanger. Everything is fine!

The problem occurs when i try to enforce a resourcequota for the namespace. The ResourceQuota looks like this:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: mem-cpu-demo
spec:
  hard:
    limits.cpu: "2"
    limits.memory: 2Gi

When I delete and recreate the pod it will not be created. The resourcequota will result in the following error:

Error from server (Forbidden): error when creating "test-pod.yml": pods "test-pod" is forbidden: failed quota: mem-cpu-demo: must specify limits.cpu

In other words, the resourcequota is applied before LimitRanger so it does not let me create pods without a specified limit.

Is there a way to enforce LimitRanger first and then the ResourceQuota? How do you apply them to your namespaces?

I would like to have developers that do not specify limits in the pod definition to be able to acquire the defaults while enforcing the resource quota as well.

-- taggelos
kubernetes
limit

1 Answer

6/4/2020

TL;DR:

Error from server (Forbidden): error when creating "test-pod.yml": pods "test-pod" is forbidden: failed quota: mem-cpu-demo: must specify limits.cpu

  • You didn't set a default limit for CPU, according to ResourceQuota Docs:

    If quota is enabled in a namespace for compute resources like cpu and memory, users must specify requests or limits for those values; otherwise, the quota system may reject pod creation.

  • This is why the pod is not being created. Add a cpu-limit.yaml:

apiVersion: v1
kind: LimitRange
metadata:
  name: cpu-limit-range
  namespace: test
spec:
  limits:
  - default:
      cpu: 1
    defaultRequest:
      cpu: 0.5
    type: Container
  • The limitRanger injects the defaults at container runtime, and yes, it injects the default values prior to the ResourceQuota validation.

  • Other minor issue that I found, is that not all your yamls contains the namespace: test line under metadata, that's important to assign the resources to the right namespace, I fixed it on the example below.

Reproduction:

  • Created namespace, applied first the mem-limit and quota, as you mentioned:
$ kubectl create namespace test
namespace/test created

$ cat mem-limit.yaml 
apiVersion: v1
kind: LimitRange
metadata:
  name: mem-limit-range
  namespace: test
spec:
  limits:
  - default:
      memory: 512Mi
    defaultRequest:
      memory: 256Mi
    type: Container

$ cat quota.yaml 
apiVersion: v1
kind: ResourceQuota
metadata:
  name: mem-cpu-demo
  namespace: test
spec:
  hard:
    limits.cpu: "2"
    limits.memory: 2Gi

$ kubectl apply -f mem-limit.yaml 
limitrange/mem-limit-range created

$ kubectl apply -f quota.yaml 
resourcequota/mem-cpu-demo created

$ kubectl describe resourcequota -n test
Name:          mem-cpu-demo
Namespace:     test
Resource       Used  Hard
--------       ----  ----
limits.cpu     0     2
limits.memory  0     2Gi

$ kubectl describe limits -n test
Name:       mem-limit-range
Namespace:  test
Type        Resource  Min  Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---  ---  ---------------  -------------  -----------------------
Container   memory    -    -    256Mi            512Mi          -
  • Now if I try to create the pod:
$ cat pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  namespace: test
spec:
  containers:
  - name: test-pod-ctr
    image: redis

$ kubectl apply -f pod.yaml 
Error from server (Forbidden): error when creating "pod.yaml": pods "test-pod" is forbidden: failed quota: mem-cpu-demo: must specify limits.cpu
  • Same error you faced, because there is no default limits for CPU set. We'll create and apply it:
$ cat cpu-limit.yaml 
apiVersion: v1
kind: LimitRange
metadata:
  name: cpu-limit-range
  namespace: test
spec:
  limits:
  - default:
      cpu: 1
    defaultRequest:
      cpu: 0.5
    type: Container

$ kubectl apply -f cpu-limit.yaml 
limitrange/cpu-limit-range created

$ kubectl describe limits cpu-limit-range -n test
Name:       cpu-limit-range
Namespace:  test
Type        Resource  Min  Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---  ---  ---------------  -------------  -----------------------
Container   cpu       -    -    500m             1              -
  • Now with the cpu limitRange in action, let's create the pod and inspect it:
$ kubectl apply -f pod.yaml 
pod/test-pod created

$ kubectl describe pod test-pod -n test
Name:         test-pod
Namespace:    test
Status:       Running
...{{Suppressed output}}...

    Limits:
      cpu:     1
      memory:  512Mi
    Requests:
      cpu:        500m
      memory:     256Mi
  • Our pod was created with the enforced limitRange.

If you have any question let me know in the comments.

-- Will R.O.F.
Source: StackOverflow