I am getting started with Kubernetes and I am trying to learn more about the liveness probe.
Some docs and articles tell the default value of failureThreshold
is 3 times. And when you do not specify failureThreshold, Kubernetes will make 3 probes before restarting the container.
My question is, how many times does kubelet restarts the pod container?
Here is a sample livenessprobe-execaction.yaml:
apiVersion: v1
kind: Pod
metadata:
labels:
test: liveness
name: liveness-exec
spec:
containers:
- name: liveness
image: k8s.gcr.io/busybox
args: # command to be executed when the container starts
- /bin/sh
- -c
- touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600 # during first 30 seconds there will be a file and cat will return success, when removed, a failure
livenessProbe:
exec:
command: # in the first probe there will be a file within 30 seconds, and no errors
# and after 35 seconds a nex probe is done but the file is gone, and error will show up and machine will be restarted
# several restarts should happen or restarts only 3 times?
- cat
- /tmp/healthy
initialDelaySeconds: 5 # kubelet waits 5 seconds before first probe
periodSeconds: 5 # kubelet checks every 5 seconds
#failureThreshold: 3 is the default number of times, after the 3rd liveness probe the container is restarted? forever?
After creating the pod:
$ kubectl apply -f livenessprobe-execaction.yaml
And running a watch:
$ kubectl get pod liveness-exec --watch
The output is:
NAME READY STATUS RESTARTS AGE
liveness-exec 1/1 Running 0 4s
liveness-exec 1/1 Running 1 75s
liveness-exec 1/1 Running 2 2m29s
liveness-exec 1/1 Running 3 3m44s
liveness-exec 1/1 Running 4 5m
liveness-exec 1/1 Running 5 6m14s
liveness-exec 0/1 CrashLoopBackOff 5 7m29s
liveness-exec 1/1 Running 6 8m57s
liveness-exec 1/1 Running 7 10m
liveness-exec 0/1 CrashLoopBackOff 7 11m
liveness-exec 1/1 Running 8 16m
liveness-exec 1/1 Running 9 17m
liveness-exec 0/1 CrashLoopBackOff 9 18m
It depends on the restartPolicy in the PodSpec.
A PodSpec has a restartPolicy
field with possible values Always
, OnFailure
, and Never
. The default value is Always
. restartPolicy
applies to all Containers in the Pod. restartPolicy
only refers to restarts of the Containers by the kubelet on the same node. Exited Containers that are restarted by the kubelet are restarted with an exponential back-off delay (10s, 20s, 40s …) capped at five minutes, and is reset after ten minutes of successful execution
Thanks for the response. I have seen some similar answers but that does not explain quite well how many times a pod would be restarted. The answer is, as of today, it would restart forever in the default mode "Always".
It seems there is no way to set the max number of restarts.
You either restart "always", "on failure" or "never. While in the always and on failure mode, the 5 minutes is just the maximum wait time between retries to start the container.
"Never" would not restart the container.
So, there is no option yet that helps us to set the maximum number of retries.
To quote Dan Wahlin, which has a course on this topic:
First off, the 5 minute number is the max cap time between restarts. If you run it again you'll see that it definitely starts slowing down the restarts a lot (see below) and will reach a point where it only tries every 5 minutes. It sticks with that number unless the container is healthy (in which case it would reset things).
As far as the max number of restarts (which is what your question was really about I now realize), it looks like they're still working on that feature from what I can tell on the github site. I'm not normally having to deal with this scenario (normally it'd be a cluster admin) but if I come across anything else I'll let you know.
https://github.com/kubernetes/kubernetes/issues/49466 https://github.com/kubernetes/kubernetes/pull/79334
Looks like this is the specific pull request that will address it: