Configure the master api server to check cordon node and destroy if it has no jobs running

12/17/2018

Team,

We need to roll out some drivers on worker nodes of a K8s cluster and our flow is as below:

  1. cordon node [no more scheduling]
  2. wait for jobs to complete
  3. destroy

Is there a way I can automate this using K8s options itself instead of writing some bash script to do those checks every time because we don't know when pods would complete. So, can we configure the master API server to check cordon node and destroy if it has no jobs running?

-- AhmFM
kube-apiserver
kubectl
kubernetes

1 Answer

12/17/2018

You can write your own application either using the Go Client, Python Client, or Java Client and basically do this:

$ kubectl apply -f yourjob.yaml
$ kubectl cordon <nodename>
$ kubectl wait --for=condition=complete job/myjob
$ kubectl drain <nodename>
# Terminate your node if drain returns successfully

If this is a frequent pattern, you could also probably leverage a custom controller (operator) with a custom resource definition (CRD) to do that. You will have to embed the code of your application that talks to the API server.

-- Rico
Source: StackOverflow