I have a Kubernetes cluster, and I am studying the behaviour of Kubernetes pod in case of failure.
So in my single master 2 worker node Kubernetes cluster, that has one StatefulSet deployed. If I inject a failure(intentionally) by changing the image of the pod in the StatefulSet spec to a non-existent version and considering the strategy Type is RollingUpdate, I see pod keeps trying for hours/days to create a container and doesn't stop this effort. Is there a configuration in Kubernetes that will let K8s take an action in such scenario?
sample-0 1/1 Running 0 23h worker1
sample-1 0/1 ImagePullBackOff 0 23h worker2 The back-off is unfortunately not tunable.
From Kubernetes official documentation:
The BackOff part indicates that Kubernetes will keep trying to pull the image, with an increasing back-off delay.
Kubernetes raises the delay between each attempt until it reaches a compiled-in limit, which is 300 seconds (5 minutes).
Reference: https://kubernetes.io/docs/concepts/containers/images/#imagepullbackoff
https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy