I was experimenting with the default-not-ready-toleration-seconds in Kubernetes to lower the time before pods get terminated on a NotReady node in the cluster. However I noticed that the CRI (Docker) will of course not kill the running container by itself if Kubelet is down. Therefore when Kubernetes decide to terminate the pod, the container will still run if it is unable to communicate with Kubelet and you will end up with 2 containers running.
My question is, is there a way to make the CRI independently make the decision to kill a container if Kubelet have not been running for a set timeout?
# docker ps | grep busybox
5cb58ea42982 busybox "tail -f /dev/null" 45 hours ago Up 45 hours
# systemctl stop kubelet
# kubectl get pod busybox-test-busybox-69b844bd79-wsdvp
NAME READY STATUS
busybox-test-busybox-69b844bd79-wsdvp 1/1 Terminating
### WAIT WAIT WAIT...
# docker ps | grep busybox
5cb58ea42982 busybox "tail -f /dev/null" 45 hours ago Up 45 hoursYou can and cannot do that, basically, kubelet is the captain on the node and instructs docker to create containers based on instructions received from the Kubernetes API server.
If the kubelet is disconnected from the master, the node is marked as not reachable after certain intervals of time.
Now kubelet is the owner of the node but without any instructions, it does not know what to do with pods(as kubelet sees them) on the node kubelet will keep the running pods in running state or can create new pods from staticPodPath
Let's say kubelet also dies. Now no one is there to instruct the docker about containers.
Node is just like a machine with Docker installed, and containers running on it. Docker is not aware that its masters have died. It will keep running the containers.
Now for your question:
My question is, is there a way to make the CRI independently make the decision to kill a container if Kubelet have not been running for a set timeout?You can create a script that runs on the node and checks the status of the kubelet service. and if service is down kill all running containers.