K
Q

Why does not Kubernetes pod graceful stop?

11/7/2018

I have encountered a problem that does not stop immediately even if I delete pod.

What should be fixed in order to terminate normally?

manifest file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cmd-example
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cmd-example
  template:
    metadata:
      labels:
        app: cmd-example
    spec:
      terminationGracePeriodSeconds: 30
      containers:
      - name: cmd-container
        image: alpine:3.8
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        command: ["/bin/sh"]
        args: ["-c", "while true; do exec sleep 100;done"]

Reproduction procedure

  1. create deployment.
    $ kubectl apply -f deployments.yaml
  2. delete deployment.
    kubectl delete-f 020-deployments.yaml

kubectl get po -woutput is.

cmd-example-5cccf79598-zpvmz   1/1       Running   0         2s
cmd-example-5cccf79598-zpvmz   1/1       Terminating   0         6s
cmd-example-5cccf79598-zpvmz   0/1       Terminating   0         37s
cmd-example-5cccf79598-zpvmz   0/1       Terminating   0         38s
cmd-example-5cccf79598-zpvmz   0/1       Terminating   0         38s

This should finish faster.
It took about 30 seconds to complete. Perhaps it is due to SIGKILL at the time of terminationGracePeriodSeconds 30s.
Why is not pod cleanup immediately with SIGTERM?

What should be fixed?

Environment

I confirmed it in the following environment.

  • Docker for Mac:18.06.1-ce,Kubernetes :v1.10.3
  • Docker for Windows:18.06.1-ce,Kubernetes :v1.10.3
  • Google Kubernete Engine:1.11.2-gke.15
-- Hiroki Matsumoto
kubernetes

Similar Questions

Ansible playbook wait until all pods running
Advice on setting pod memory request size
Tainting in k8s
Updating a Pod with its Secret
how to see the pod and veth relationship in kubernetes
Escape k8s nodePort range

3 Answers

11/8/2018

Your pod literally does nothing. If you just want something where you can do occasional interactive debugging "inside the cluster", consider kubectl run to get a one-off interactive container

kubectl run --rm -it --name debug --image alpine:3.8

In terms of the command your pod spec is trying to run, rewriting it in shell script form:

#!/bin/sh
# Forever:
while true
do
  # Replace this shell with a process that sleeps for
  # 100 ms, then exits
  exec sleep 100
  # The shell no longer exists and you'll never get here
done

I'm not clear what the pod is trying to do, but it at least won't exit if you remove the exec. (It will still sit in an idle loop forever.)

-- David Maze
Source: StackOverflow
11/8/2018

Cause of a problem

This shell is that it does not stop even if it accepts the signal of SIGTERM.

Solution

Using the trap command.

Changed place

    command: ["/bin/sh"]
    args: ["-c", "trap 'exit 0' 15;while true; do exec sleep 100 & wait $!; done"]

Result

after delete, pod was cleaned up as soon!

img-example-d68954677-mwsqp   1/1       Running   0         2s
img-example-d68954677-mwsqp   1/1       Terminating   0         8s
img-example-d68954677-mwsqp   0/1       Terminating   0         10s
img-example-d68954677-mwsqp   0/1       Terminating   0         11s
img-example-d68954677-mwsqp   0/1       Terminating   0         11s
-- Hiroki Matsumoto
Source: StackOverflow
11/8/2018

Hiroki Matsumoto, the pod termination is behaving just like it was designed to behave. As you can find in documentation section on Pods:

Because pods represent running processes on nodes in the cluster, it is important to allow those processes to gracefully terminate when they are no longer needed (vs being violently killed with a KILL signal and having no chance to clean up).

Long story short (based on official documentation)

1) When you run kubectl delete -f deployments.yaml you send a command with time of grace period (by default 30 seconds)

2) when you run kubectl get pods you can see it has terminating state

3) Kubelet sees this state and Pod starts to shutdown.

4) After the grace period is over, if there is any processes still running it is killed with SIGKILL

So to delete a pod immediately you have to lower the grace period to 0 and run a forced/immediate deletion:

kubectl delete -f deployments.yaml --grace-period=0 --force and this results in an instant deletion.

-- aurelius
Source: StackOverflow