Does Kubelet repeatedly restart the pod by default?

4/11/2020

I am getting started with Kubernetes and I am trying to learn more about the liveness probe.

Some docs and articles tell the default value of failureThreshold is 3 times. And when you do not specify failureThreshold, Kubernetes will make 3 probes before restarting the container.

My question is, how many times does kubelet restarts the pod container?

Here is a sample livenessprobe-execaction.yaml:

apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-exec
spec:
  containers:
  - name: liveness
    image: k8s.gcr.io/busybox
    args: # command to be executed when the container starts
    - /bin/sh
    - -c
    - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600 # during first 30 seconds there will be a file and cat will return success, when removed, a failure
    livenessProbe:
      exec:
        command: # in the first probe there will be a file within 30 seconds, and no errors
                 # and after 35 seconds a nex probe is done but the file is gone, and error will show up and machine will be restarted
                 # several restarts should happen or restarts only 3 times?
        - cat
        - /tmp/healthy
      initialDelaySeconds: 5 # kubelet waits 5 seconds before first probe
      periodSeconds: 5 # kubelet checks every 5 seconds
      #failureThreshold: 3 is the default number of times, after the 3rd liveness probe the container is restarted? forever?

After creating the pod:

$ kubectl apply -f livenessprobe-execaction.yaml

And running a watch:

$ kubectl get pod liveness-exec --watch

The output is:

    NAME            READY   STATUS    RESTARTS   AGE
    liveness-exec   1/1     Running   0          4s
    liveness-exec   1/1     Running   1          75s
    liveness-exec   1/1     Running   2          2m29s
    liveness-exec   1/1     Running   3          3m44s
    liveness-exec   1/1     Running   4          5m
    liveness-exec   1/1     Running   5          6m14s
    liveness-exec   0/1     CrashLoopBackOff   5          7m29s
    liveness-exec   1/1     Running            6          8m57s
    liveness-exec   1/1     Running            7          10m
    liveness-exec   0/1     CrashLoopBackOff   7          11m
    liveness-exec   1/1     Running            8          16m
    liveness-exec   1/1     Running            9          17m
    liveness-exec   0/1     CrashLoopBackOff   9          18m
-- Junior Mayhé
kubernetes

2 Answers

4/12/2020

It depends on the restartPolicy in the PodSpec.

A PodSpec has a restartPolicy field with possible values Always, OnFailure, and Never. The default value is Always. restartPolicy applies to all Containers in the Pod. restartPolicy only refers to restarts of the Containers by the kubelet on the same node. Exited Containers that are restarted by the kubelet are restarted with an exponential back-off delay (10s, 20s, 40s …) capped at five minutes, and is reset after ten minutes of successful execution

-- Arghya Sadhu
Source: StackOverflow

4/26/2020

Thanks for the response. I have seen some similar answers but that does not explain quite well how many times a pod would be restarted. The answer is, as of today, it would restart forever in the default mode "Always".

It seems there is no way to set the max number of restarts.

You either restart "always", "on failure" or "never. While in the always and on failure mode, the 5 minutes is just the maximum wait time between retries to start the container.

"Never" would not restart the container.

So, there is no option yet that helps us to set the maximum number of retries.

To quote Dan Wahlin, which has a course on this topic:

First off, the 5 minute number is the max cap time between restarts. If you run it again you'll see that it definitely starts slowing down the restarts a lot (see below) and will reach a point where it only tries every 5 minutes. It sticks with that number unless the container is healthy (in which case it would reset things).

As far as the max number of restarts (which is what your question was really about I now realize), it looks like they're still working on that feature from what I can tell on the github site. I'm not normally having to deal with this scenario (normally it'd be a cluster admin) but if I come across anything else I'll let you know.

https://github.com/kubernetes/kubernetes/issues/49466 https://github.com/kubernetes/kubernetes/pull/79334

Looks like this is the specific pull request that will address it:

https://github.com/kubernetes/kubernetes/pull/79334/files/db71ebf0ec9bc04c48542059ccd46a34a2abcc16#diff-e281c21e93f01ecc8cf12e4ff535b3e5

-- Junior Mayhé
Source: StackOverflow