Getting into a strange scenario where, after 2-3 days pods of an application moves into init state;
whereas none of its container has restarted.
kubectl describe
of pod gives following status about its phases.
Conditions:
Type Status
Initialized False
Ready True
ContainersReady True
PodScheduled True
As we can see all its containers are running and available but somehow, its init container started to execute again and gets stuck in some error. I checked for restart counts for individual container and it is 0. This is not one off scenario, this is reproducible every 2-3 days. I also looked at RAM and CPU spikes and nothing there.
In which scenario can this issue happen? What could trigger execution of init container without any restart or without failure of its containers.
In our case, it was an housekeeping script that was doing
docker system prune -a
and it was deleting all untagged containers. It also included our init container, may be because it was completed/stopped. I will recommend to use kubernetes GC rather than executing these low level commands in cluster.