K
Q

Question

How to force delete a Kubernetes Namespace?

4/25/2019

How do I force delete Namespaces stuck in Terminating?

Steps to recreate:

Apply this YAML

apiVersion: v1
kind: Namespace
metadata:
  name: delete-me
spec:
  finalizers:
    - foregroundDeletion

kubectl delete ns delete-me
It is not possible to delete delete-me.

The only workaround I've found is to destroy and recreate the entire cluster.

Things I've tried:

None of these work or modify the Namespace. After any of these the problematic finalizer still exists.

Edit the YAML and `kubectl apply`

Apply:

apiVersion: v1
kind: Namespace
metadata:
  name: delete-me
spec:
  finalizers:

$ kubectl apply -f tmp.yaml 

namespace/delete-me configured

The command finishes with no error, but the Namespace is not udpated.

The below YAML has the same result:

apiVersion: v1
kind: Namespace
metadata:
  name: delete-me
spec:

`kubectl edit`

kubectl edit ns delete-me, and remove the finalizer. Ditto removing the list entirely. Ditto removing spec. Ditto replacing finalizers with an empty list.

$ kubectl edit ns delete-me 

namespace/delete-me edited

This shows no error message but does not update the Namespace. kubectl editing the object again shows the finalizer still there.

`kubectl proxy &`

kubectl proxy &
curl -k -H "Content-Type: application/yaml" -X PUT --data-binary @tmp.yaml http://127.0.0.1:8001/api/v1/namespaces/delete-me/finalize

As above, this exits successfully but does nothing.

Force Delete

kubectl delete ns delete-me --force --grace-period=0

This actually results in an error:

warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
Error from server (Conflict): Operation cannot be fulfilled on namespaces "delete-me": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.

However, it doesn't actually do anything.

Wait a long time

In the test cluster I set up to debug this issue, I've been waiting over a week. Even if the Namespace might eventually decide to be deleted, I need it to be deleted faster than a week.

Make sure the Namespace is empty

The Namespace is empty.

$ kubectl get -n delete-me all

No resources found.

`etcdctl`

$ etcdctl --endpoint=http://127.0.0.1:8001 rm /namespaces/delete-me

Error:  0:  () [0]

I'm pretty sure that's an error, but I have no idea how to interpret that. It also doesn't work. Also tried with --dir and -r.

`ctron/kill-kube-ns`

There is a script for force deleting Namespaces. This also does not work.

$ ./kill-kube-ns delete-me

Killed namespace: delete-me

$ kubectl get ns delete-me 

NAME        STATUS        AGE
delete-me   Terminating   1h

`POST`ing the edited resource to /finalize

Returns a 405. I'm not sure if this is the canonical way to POST to /finalize though.

Links

This appears to be a recurring problem and none of these resources helped.

Kubernetes bug

-- Will Beason

kubernetes

kubernetes-namespace

1 Answer

4/25/2019

The kubectl proxy try is almost correct, but not quite. It's possible using JSON instead of YAML does the trick, but I'm not certain.

The JSON with an empty finalizers list:

~$ cat ns.json

{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "delete-me"
  },
  "spec": {
    "finalizers": []
  }
}

Use curl to PUT the object without the problematic finalizer.

~$ curl -k -H "Content-Type: application/json" -X PUT --data-binary @ns.json http://127.0.0.1:8007/api/v1/namespaces/delete-me/finalize

{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "delete-me",
    "selfLink": "/api/v1/namespaces/delete-me/finalize",
    "uid": "0df02f91-6782-11e9-8beb-42010a800137",
    "resourceVersion": "39047",
    "creationTimestamp": "2019-04-25T17:46:28Z",
    "deletionTimestamp": "2019-04-25T17:46:31Z",
    "annotations": {
      "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"name\":\"delete-me\"},\"spec\":{\"finalizers\":[\"foregroundDeletion\"]}}\n"
    }
  },
  "spec": {

  },
  "status": {
    "phase": "Terminating"
  }
}

The Namespace is deleted!

~$ kubectl get ns delete-me

Error from server (NotFound): namespaces "delete-me" not found

-- Will Beason

Source: StackOverflow

KQ

How to force delete a Kubernetes Namespace?

Steps to recreate:

Things I've tried:

Edit the YAML and kubectl apply

kubectl edit

kubectl proxy &