Is there a way to configure Kube-Scheduler to use only certain percentage of capacity on nodes? For example, I want scheduler to stop scheduling pods on nodes that have 70% of their memory and CPU resources utilized.
Based on the scheduling policies documentation, PodFitsResources predicate seems to fit closest to my requirement. However, is there a way to configure this predicate such that after 70% resources are consumed this node is unavailable during scheduler filtering?
There is way to do that with setting Kubelet parameters via a config file using --kube-reserved flag.
kube-reservedis meant to capture resource reservation for kubernetes system daemons like thekubelet,container runtime,node problem detector, etc.
With setting this flag you will reserve appropriate amount of resources for system (in your use case that would 30%) which will leave with the 70% available resources for your running pods.