Kubernetes maxPods int32 also include its defaults pods

3/13/2019

I want to set the maximum allowed pods (that my application need/create) on any kubernetes node as '3' (in Default namespace).

When I set maxPods: 3 in /var/lib/kubelet/config.yaml, seems like it also includes the pods cerated by kubernetes internally(namespace: 'kube-system') in this number. In other words unless I set maxPods to 7 (on the node which is also Master) I cannot get any application pod created in default namespace.

My node ( I set maxPods on node1.co.com as '8' and I can only get 2 application pods in default namespace):

$ kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, cap: .status.capacity}'
{
  "name": "node1.co.com",
  "cap": {
    "cpu": "2",
    "ephemeral-storage": "8649700Ki",
    "hugepages-2Mi": "0",
    "memory": "8007036Ki",
    "pods": "8"
  }
}
{
  "name": "node2.co.com",
  "cap": {
    "cpu": "2",
    "ephemeral-storage": "8649700Ki",
    "hugepages-2Mi": "0",
    "memory": "8009324Ki",
    "pods": "9"
  }
}

Any additional pods are in pending state:

$ kubectl create -f single-node-deployement4.yaml
$ kubectl get pods -o=wide --all-namespaces | grep Pending

default       single-node-deployment4-585466c56c-sszdz                  0/1     Pending     0          17s     <none>           <none>                                  <none>           <none>

$ kubectl get pods -o=wide --all-namespaces | grep OutOf

kube-system   coredns-86c58d9df4-7gb5d                                        0/1     OutOfpods   0          7d19h   <none>           node1.co.com   <none>           <none>
kube-system   coredns-86c58d9df4-t6rcj                                        0/1     OutOfpods   0          7d19h   <none>           node1.co.com   <none>           <none>

Is number of kubernetes internal pods constant and I can assume to have maximum of three pods, I can set 'maxPods: 9' or is there a better way to set maxPods?

-- techsurvivor
kubeadm
kubernetes

1 Answer

3/19/2019

I think you could achieve the same result

  • limit #Pods per namespace
  • do not schedule on master node

by using these Kubernetes policies/features together:

  1. ResourceQuota (object count quota)
  2. Node Taint

Example:

Create resource quota based on object count:

  • kubectl create namespace limited

# Source: object-count-quota.yaml apiVersion: v1 kind: ResourceQuota metadata: name: object-counts spec: hard: pods: "3"

  • kubectl create -f ./object-count-quota.yaml -n limited

  • kubectl describe resourcequota -n limited

Make master node unschedulable:

  • kubectl taint nodes master-node key=value:NoSchedule
-- Nepomucen
Source: StackOverflow