What is my goal:
I have a research task where I'm using Kubernetes Java client to subscribe to Kubernetes API and receive events about all pods from all namespaces (create, modify, delete). My goal is to make some analysis on the basis of container statuses from the response.
What I am doing:
I have a test cluster with Kubernetes 1.14. The cluster is running on some nodes and spinning a bunch of pods. For testing purposes, I've chosen 2 pods running in a replica set. Each pod has 1 container. Restarting policy is "Always". Anti-affinity feature is turned on in order not to run 2 copies on the same node. Time to time I delete a randomly chosen pod from the replica set and check the response. I expect that a container inside the pod switches from Running state to Terminated state, the pod terminates, and a new pod starts.
What I have faced:
Mostly it works as expected. But sometimes a container switches from Running state to Waiting state with the reason "ContainerCreating". After a while, it was revealed that it happens only if a newly created pod is assigned to exactly the same node where a deleted pod is still terminating.
Question:
Why does container not change its state to Terminated state? Is there any purpose to change the state another way?