Pod moved to init state without restart

3/23/2020

Getting into a strange scenario where, after 2-3 days pods of an application moves into init state; whereas none of its container has restarted.

kubectl describe of pod gives following status about its phases.

Conditions:
  Type              Status
  Initialized       False 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 

As we can see all its containers are running and available but somehow, its init container started to execute again and gets stuck in some error. I checked for restart counts for individual container and it is 0. This is not one off scenario, this is reproducible every 2-3 days. I also looked at RAM and CPU spikes and nothing there.

In which scenario can this issue happen? What could trigger execution of init container without any restart or without failure of its containers.

-- Prateek Jain
kubernetes
kubernetes-pod

1 Answer

3/24/2020

In our case, it was an housekeeping script that was doing

docker system prune -a

and it was deleting all untagged containers. It also included our init container, may be because it was completed/stopped. I will recommend to use kubernetes GC rather than executing these low level commands in cluster.

-- Prateek Jain
Source: StackOverflow