There could be a lot of circumstances that will alter the work of the scheduler. There is a documentation talking about it: Pod priority and preemption.

Be aware of the fact that this features were deemed stable at version 1.14.0

From the IBM perspective please take in mind that the version 1.13.9 will be supported until 19 of February 2020!.

You are correct that pods with lower priority should be replaced with higher priority pods.

Let me elaborate on that with an example:

Example

Let's assume a Kubernetes cluster with 3 nodes (1 master and 2 nodes):

• By default you cannot schedule normal pods on master node
• The only worker node that can schedule the pods has 8GB of RAM.
• 2nd worker node has a taint that disables scheduling.

This example will base on RAM usage but it can be used in the same manner as CPU time.

Priority Class

There are 2 priority classes:

• zero-priority (0)
• high-priority (1 000 000)

YAML definition of zero priority class:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: zero-priority
value: 0
globalDefault: false
description: "This is priority class for hello pod"

globalDefault: false is used for objects that do not have assigned priority class. It will assign this class by default.

YAML definition of high priority class:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "This is priority class for goodbye pod"

To apply this priority classes you will need to invoke: $ kubectl apply -f FILE.yaml

Deployments

With above objects you can create deployments:

• Hello - deployment with low priority
• Goodbye - deployment with high priority

YAML definition of hello deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
spec:
  selector:
    matchLabels:
      app: hello
      version: 1.0.0
  replicas: 10
  template:
    metadata:
      labels:
        app: hello
        version: 1.0.0
    spec:
      containers:
      - name: hello
        image: "gcr.io/google-samples/hello-app:1.0"
        env:
        - name: "PORT"
          value: "50001"
        resources:
          requests:
            memory: "128Mi"
      priorityClassName: zero-priority

Please take a specific look on this fragment:

        resources:
          requests:
            memory: "128Mi"
      priorityClassName: zero-priority

It will limit number of pods because of the requested resources as well as it will assign low priority to this deployment.

YAML definition of goodbye deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: goodbye
spec:
  selector:
    matchLabels:
      app: goodbye
      version: 2.0.0
  replicas: 3
  template:
    metadata:
      labels:
        app: goodbye
        version: 2.0.0
    spec:
      containers:
      - name: goodbye
        image: "gcr.io/google-samples/hello-app:2.0"
        env:
        - name: "PORT"
          value: "50001"
        resources:
          requests:
            memory: "6144Mi"
      priorityClassName: high-priority

Also please take a specific look on this fragment:

        resources:
          requests:
            memory: "6144Mi"
      priorityClassName: high-priority

This pods will have much higher request for RAM and high priority.

Testing and troubleshooting

There is no enough information to properly troubleshoot issues like this. Without extensive logs of many components starting from kubelet to pods,nodes and deployments itself.

Apply hello deployment and see what happens: $ kubectl apply -f hello.yaml

Get basic information about the deployment with command:

$ kubectl get deployments hello

Output should look like that after a while:

NAME    READY   UP-TO-DATE   AVAILABLE   AGE
hello   10/10   10           10          9s

As you can see all of the pods are ready and available. The requested resources were assigned to them.

To get more details for troubleshooting purposes you can invoke:

• $ kubectl describe deployment hello
• $ kubectl describe node NAME_OF_THE_NODE

Example information about allocated resources from the above command:

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests      Limits
  --------           --------      ------
  cpu                250m (12%)    0 (0%)
  memory             1280Mi (17%)  0 (0%)
  ephemeral-storage  0 (0%)        0 (0%)

Apply goodbye deployment and see what happens: $ kubectl apply -f goodbye.yaml

Get basic information about the deployments with command: $ kubectl get deployments

NAME      READY   UP-TO-DATE   AVAILABLE   AGE
goodbye   1/3     3            1           25s
hello     9/10    10           9           11m

As you can see there is goodbye deployment but only 1 pod is available. And despite the fact that the goodbye has much higher priority, the hello pods are still working.

Why it is like that?:

$ kubectl describe node NAME_OF_THE_NODE

Non-terminated Pods:          (13 in total)
  Namespace                   Name                                        CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                   ----                                        ------------  ----------  ---------------  -------------  ---
  default                     goodbye-575968c8d6-bnrjc                    0 (0%)        0 (0%)      6Gi (83%)        0 (0%)         15m
  default                     hello-fdfb55c96-6hkwp                       0 (0%)        0 (0%)      128Mi (1%)       0 (0%)         27m
  default                     hello-fdfb55c96-djrwf                       0 (0%)        0 (0%)      128Mi (1%)       0 (0%)         27m

Take a look at requested memory for goodbye pod. It is as described above as 6Gi.

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests      Limits
  --------           --------      ------
  cpu                250m (12%)    0 (0%)
  memory             7296Mi (98%)  0 (0%)
  ephemeral-storage  0 (0%)        0 (0%)
Events:              <none>

The memory usage is near 100%.

Getting information about specific goodbye pod that is in Pending state will yield some more information $ kubectl describe pod NAME_OF_THE_POD_IN_PENDING_STATE:

Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  38s (x3 over 53s)  default-scheduler  0/3 nodes are available: 1 Insufficient memory, 2 node(s) had taints that the pod didn't tolerate.

Goodbye pod was not created because there were not enough resources that could be satisfied. But there still was some left resources for hello pods.

There is possibility for a situation that will kill lower priority pods and schedule higher priority pods.

Change the requested memory for goodbye pod to 2304Mi. It will allow scheduler to assign of all required pods (3):

        resources:
          requests:
            memory: "2304Mi"

You can delete the previous deployment and apply new one with memory parameter changed.

Invoke command: $ kubectl get deployments

NAME      READY   UP-TO-DATE   AVAILABLE   AGE
goodbye   3/3     3            3           5m59s
hello     3/10    10           3           48m

As you can see all of the goodbye pods are available.

Hello pods got reduced to make space for pods with higher priority (goodbye).