K
Q

Foreground cascading deletion not working as documentation suggests

2/10/2020

I have an instance of CRD A and an instance of CRD B. B has an owner reference to A with BlockOwnerDeletion set to true. Both instances have a finalizer.

When I delete A a DeletionTimestamp is set but no foregroundDeletion finalizer is present. Even when I explicitly add the foregroundDeletion finalizer to A before deleting. This all happens before B has been deleted.

The documentation says:

In foreground cascading deletion, the root object first enters a “deletion in progress” state. In the “deletion in progress” state, the following things are true:

  • The object is still visible via the REST API
  • The object’s deletionTimestamp is set
  • The object’s metadata.finalizers contains the value “foregroundDeletion”.

Once the “deletion in progress” state is set, the garbage collector deletes the object’s dependents. Once the garbage collector has deleted all “blocking” dependents (objects with ownerReference.blockOwnerDeletion=true), it deletes the owner object.

Note that in the “foregroundDeletion”, only dependents with ownerReference.blockOwnerDeletion=true block the deletion of the owner object. Kubernetes version 1.7 added an admission controller that controls user access to set blockOwnerDeletion to true based on delete permissions on the owner object, so that unauthorized dependents cannot delay deletion of an owner object.

If an object’s ownerReferences field is set by a controller (such as Deployment or ReplicaSet), blockOwnerDeletion is set automatically and you do not need to manually modify this field

This, to me, suggests that if B has an owner reference to A with BlockOwnerDeletion==true the finalizer foregroundDeletion should be added to A.

Am I completely misunderstanding this?

-- granra
kubernetes

Similar Questions

Not able to create ingress resource in kubernetes with service in different namespace
Does Google Kubernetes Engine support custom node images and/or 10Gbps networking?
How to kill a multi-container pod if one container fails?
How to change identity provider callback URL in Keycloak

1 Answer

2/27/2020

According to:

This, to me, suggests that if B has an owner reference to A with BlockOwnerDeletion==true the finalizer foregroundDeletion should be added to A.

You are understanding it correctly according to documentation.

If you think that observed behavior is different from what the official documentation says it should be reported as an issue on github page.

Link: Github.com: Kubernetes issues

I've tried to reproduce some parts of it on basic examples and here's what I found:

Taking:

  • basic pod with NGINX image as a parent
  • basic pod with NGINX image as a child

Steps:

  • Create a parent
  • Copy uid of parent to ownerReference of child and create it

  • Check the behavior:

    • kubectl delete
    • default curl
    • curl with foregroundDeletion option

Create a parent

Here is example YAML definition of basic pod:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-owner
  namespace: default
spec:
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: nginx-owner

Copy uid of parent to ownerReference of child and create it

Get the uid of parent pod by running command:

$ kubectl get pods nginx-owner -o yaml | grep uid | cut -d ":" -f 2

Paste it inside of YAML definition of child

apiVersion: v1
kind: Pod
metadata:
  name: nginx-child
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    kind: Pod
    name: nginx-owner
    uid: HERE!
spec:
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: nginx-child

Save and run it.

Check the behavior

Check if both pods are running and if child has ownerReference to parent

kubectl delete

Deleting parent pod with $ kubectl delete pod nginx-owner deletes:

  • nginx-owner (parent)
  • nginx-child (child) without any issues.

Annotations in parent pod right after deleting it:

  deletionGracePeriodSeconds: 30
  deletionTimestamp: "2020-02-27T14:17:48Z"

default curl

Assuming access to Kubernetes API on localhost and port 8080 with deletion command: $ curl -X DELETE localhost:8080/api/v1/default/pod/nginx-owner

Deleting parent pod by API access with default options:

  • deletes parent first
  • deletes child second

The same annotations are present in parent as in kubectl delete

  deletionGracePeriodSeconds: 30
  deletionTimestamp: "2020-02-27T15:33:18Z"

curl with foregroundDeletion option

Assuming access to Kubernetes API on localhost and port 8080 with modified deletion command to include foregroundDeletion option:

$ curl -X DELETE localhost:8080/api/v1/namespaces/default/pods/nginx-owner -d '{"kind":"DeleteOptions","apiVersion":"v1","propagationPolicy":"Foreground"}' -H "Content-Type: application/json"

curl based on Kubernetes.io: Garbage collection

Will:

  • force parent into Terminating state
  • do not delete child (Running state)
  • stuck parent pod in Terminating state

Annotations for parent pod:

  deletionGracePeriodSeconds: 30
  deletionTimestamp: "2020-02-27T15:44:52Z"
  finalizers:
  - foregroundDeletion

It added finalizers argument with foregroundDeletion but pod does not get deleted for a reason unknown to me.

EDIT:

I found the error why foreGroundDeletion wasn't working. There was an issue with:

  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    kind: Pod
    name: nginx-owner
    uid: HERE!

It should be:

  ownerReferences:
  - apiVersion: v1
    blockOwnerDeletion: true
    kind: Pod
    name: nginx-owner
    uid: HERE!
-- Dawid Kruk
Source: StackOverflow