K
Q

Graceful pod termination

9/14/2016

I need to let the container to run 5 minutes after the kubectl ' termination. It needs to do some work before it's destroyed. It seems that kubernetes contains exactly what I need:

terminationGracePeriodSeconds: 300

so I defined it within my yaml. I've updated running RCs, delete current pods so new ones were created and now I can see that a pod contains exactly this setting via get pod xyz -o=yaml.

Unfortunately, when I tried to do rolling-update, the original pod was killed after exactly 1 minute, not after 5 minutes. I does ssh to the target machine and I could see that docker termineted the container after this time.

I tried to do some investigation how the feature works. I finally found the documentation to kubectl delete where there is a notion about graceful termination period:

http://kubernetes.io/docs/user-guide/pods/

By default, all deletes are graceful within 30 seconds. The kubectl delete command supports the --grace-period= option which allows a user to override the default and specify their own value. The value 0 indicates that delete should be immediate, and removes the pod in the API immediately so a new pod can be created with the same name. On the node pods that are set to terminate immediately will still be given a small grace period before being force killed

So I took one pod, nginx, and try to delete it with grace-period=30. It turned out, that original pod was immediately delete and get pods showed that new one was being started.

So no 30 seconds. What am I doing wrong? It seems that all pods kubernetes does not take these values into account. Note that I'm using kubernetes v1.2.2

I also found this issue https://github.com/kubernetes/kubernetes/issues/24695 where the reporter had same problem and he solved it in the same fashion. So e.g. 300 seconds is not too much for kubernetes.

-- Martin Podval
kubernetes

Similar Questions

Spinnaker with Kubernetes: Not able to connect the docker hub repository
Is change of kubernetes limits applied to running pod?
Mapping incoming port in kubernetes service to different port on docker container
Kubernetes vSphere Setup IPV4 issue

2 Answers

4/11/2017

https://pracucci.com/graceful-shutdown-of-kubernetes-pods.html may this can help you.

There’re some circumstances where a SIGTERM violently kill the application, vanishing all your efforts to gracefully shutdown it. Nginx, for example, quickly exit on SIGTERM.

-- tsjsdbd
Source: StackOverflow
9/14/2016

You may be able to set magic sleep in 'preStop' hook.　This hook will be extecuted prior to kubectl sending SIGTERM to your container.

http://kubernetes.io/docs/user-guide/production-pods/#lifecycle-hooks-and-termination-notice

something like:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx
spec:
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sleep","300"]
-- everpeace
Source: StackOverflow