Kubernetes gracefull shut down of Springboot Tomcat application

10/11/2019

I am running a java spring boot application on tomcat in a kuberentes cluster, I am using a lifecycle hook to shutdown tomcat before the container gets terminated. Also defining a termination grace period of 30 secs.

expected behaviour: tomcat shutsdown gracefully when I kill a pod or delete a deployment.

actual behaviour: tomcat shutsdown abruptly.

lifecycle:
          preStop:
            exec:
              command:
              - /usr/local/tomcat/bin/catalina.sh

Can anyone help please?

-- Lakshmi Reddy
kubernetes
tomcat

2 Answers

10/11/2019

This is a common mistake about graceful period concept. People think if they set a graceful period of say 30 seconds, when deleting the pod, it is going to wait for 30 seconds, which is wrong.

The grace period is actually for killing a pod, not to keep it alive. If you have an app that shuts down in 1 second, you can set any grace period, but the pod is going to die in 1 second. It would kill the pod (by sending SIGKILL), if it passes the grace period.

So, you would set a grace period to not have a pod in terminating state forever (and kill it after the certain amount of time), rather then keep the pod running until certain amount of time after deleting it. Hope this makes sense.

And you preStop probably runs, if everything is set properly, but it will depend on what are you doing to see the results. You can't see logs, for example, unless you have made them persistent.

-- suren
Source: StackOverflow

10/14/2019

In Kubernetes, the preStop hook is executed before the pod is terminated.

For the above use case, tomcat process has to be shutdown gracefully before the pod is terminated when a pod / deployment is deleted.

The following sample pod definition works, by graceful shutdown of the tomcat server, before pod termination. Note: The container in this definition is FROM tomcat:9.0.19-jre8

apiVersion: v1 
kind: Pod 
metadata:
  name: demopod
spec:
  containers:
  - image: demo:web
    name: demo-container
    ports:
    - containerPort: 8080
    lifecycle:
      postStart:
        exec:
          command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
      preStop:
        exec:
          command: ["/bin/sh", "-c", "echo Stopping server > /usr/share/message1; sleep 10; sh /usr/local/tomcat/bin/catalina.sh stop"]

In the above definition file, we run three commands (change as per need).

  1. Write a shutdown message into /usr/share/message1

  2. sleep 10 (Just to give time to view pod / container logs)

  3. Run Catalina.sh stop script to stop the tomcat server.

To Debug: For PreStop, if the handler fails, an event is broadcasted, the FailedPreStopHook event. You can see these events by running kubectl describe pod <pod_name>

Shutdown logs: Valid shutdown logs

-- Muralidharan.rade
Source: StackOverflow