Pods stuck in Terminating status

2/17/2016

I tried to delete a ReplicationController with 12 pods and I could see that some of the pods are stuck in Terminating status.

My Kubernetes cluster consists of one control plane node and three worker nodes installed on Ubuntu virtual machines.

What could be the reason for this issue?

NAME        READY     STATUS        RESTARTS   AGE
pod-186o2   1/1       Terminating   0          2h
pod-4b6qc   1/1       Terminating   0          2h
pod-8xl86   1/1       Terminating   0          1h
pod-d6htc   1/1       Terminating   0          1h
pod-vlzov   1/1       Terminating   0          1h
-- Dimuthu
kubernetes

10 Answers

1/23/2019

I stumbled upon this recently to free up resource in my cluster. here is the command to delete them all.

kubectl get pods --all-namespaces | grep Terminating | while read line; do 
pod_name=$(echo $line | awk '{print $2}' ) name_space=$(echo $line | awk 
'{print $1}' ); kubectl delete pods $pod_name -n $name_space --grace-period=0 --force; 
done

hope this help someone who read this

-- rrr
Source: StackOverflow

7/4/2016

You can use following command to delete the POD forcefully.

kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE>
-- Nitin
Source: StackOverflow

1/19/2017

Force delete the pod:

kubectl delete pod --grace-period=0 --force --namespace <NAMESPACE> <PODNAME>

The --force flag is mandatory.

-- Joan
Source: StackOverflow

2/27/2018

Delete the finalizers block from resource (pod,deployment,ds etc...) yaml:

"finalizers": [
  "foregroundDeletion"
]
-- Roee Rakovsky
Source: StackOverflow

2/18/2016

Practical answer -- you can always delete a terminating pod by running:

kubectl delete pod NAME --grace-period=0

Historical answer -- There was an issue in version 1.1 where sometimes pods get stranded in the Terminating state if their nodes are uncleanly removed from the cluster.

-- Alex Robinson
Source: StackOverflow

3/9/2019

I found this command more straightforward:

for p in $(kubectl get pods | grep Terminating | awk '{print $1}'); do kubectl delete pod $p --grace-period=0 --force;done

It will delete all pods in Terminating status in default namespace.

-- belabrinel
Source: StackOverflow

5/9/2019

In my case the --force option didn't quite work. I could still see the pod ! It was stuck in Terminating/Unknown mode. So after running

kubectl delete pods <pod> -n redis --grace-period=0 --force

I ran

kubectl patch pod <pod> -p '{"metadata":{"finalizers":null}}'
-- sh0umik
Source: StackOverflow

3/29/2018

If --grace-period=0 is not working then you can do:

kubectl delete pods <pod> --grace-period=0 --force
-- Paul Ma
Source: StackOverflow

7/14/2019

The original question is "What could be the reason for this issue?" and the answer is discussed at https://github.com/kubernetes/kubernetes/issues/51835 & https://github.com/kubernetes/kubernetes/issues/65569 & see https://www.bountysource.com/issues/33241128-unable-to-remove-a-stopped-container-device-or-resource-busy

Its caused by docker mount leaking into some other namespace.

You can logon to pod host to investigate.

minikube ssh
docker container ps | grep <id>
docker container stop <id> 
-- noelmcloughlin
Source: StackOverflow

9/19/2018

I stumbled upon this recently when removing rook ceph namespace - it got stuck in Terminating state.

The only thing that helped was removing kubernetes finalizer by directly calling k8s api with curl as suggested here.

  • kubectl get namespace rook-ceph -o json > tmp.json
  • delete kubernetes finalizer in tmp.json (leave empty array "finalizers": [])
  • run kubectl proxy in another terminal for auth purposes and run following curl request to returned port
  • curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json 127.0.0.1:8001/k8s/clusters/c-mzplp/api/v1/namespaces/rook-ceph/finalize
  • namespace is gone

Detailed rook ceph teardown here.

-- zub0r
Source: StackOverflow