I want to set the maximum allowed pods (that my application need/create) on any kubernetes node as '3' (in Default namespace).
When I set maxPods: 3
in /var/lib/kubelet/config.yaml
, seems like it also includes the pods cerated by kubernetes internally(namespace: 'kube-system'
) in this number. In other words unless I set maxPods to 7 (on the node which is also Master) I cannot get any application pod created in default namespace.
My node ( I set maxPods
on node1.co.com as '8' and I can only get 2 application pods in default namespace):
$ kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, cap: .status.capacity}'
{
"name": "node1.co.com",
"cap": {
"cpu": "2",
"ephemeral-storage": "8649700Ki",
"hugepages-2Mi": "0",
"memory": "8007036Ki",
"pods": "8"
}
}
{
"name": "node2.co.com",
"cap": {
"cpu": "2",
"ephemeral-storage": "8649700Ki",
"hugepages-2Mi": "0",
"memory": "8009324Ki",
"pods": "9"
}
}
Any additional pods are in pending state:
$ kubectl create -f single-node-deployement4.yaml
$ kubectl get pods -o=wide --all-namespaces | grep Pending
default single-node-deployment4-585466c56c-sszdz 0/1 Pending 0 17s <none> <none> <none> <none>
$ kubectl get pods -o=wide --all-namespaces | grep OutOf
kube-system coredns-86c58d9df4-7gb5d 0/1 OutOfpods 0 7d19h <none> node1.co.com <none> <none>
kube-system coredns-86c58d9df4-t6rcj 0/1 OutOfpods 0 7d19h <none> node1.co.com <none> <none>
Is number of kubernetes internal pods constant and I can assume to have maximum of three pods, I can set 'maxPods: 9'
or is there a better way to set maxPods
?
I think you could achieve the same result
by using these Kubernetes policies/features together:
Example:
Create resource quota based on object count:
kubectl create namespace limited
# Source: object-count-quota.yaml apiVersion: v1 kind: ResourceQuota metadata: name: object-counts spec: hard: pods: "3"
kubectl create -f ./object-count-quota.yaml -n limited
kubectl describe resourcequota -n limited
Make master node unschedulable:
kubectl taint nodes master-node key=value:NoSchedule