How to gracefully shutdown servelts in tomcat in docker container?

8/3/2018

What I found out so far:

  • A "docker stop" sends a SIGTERM to process ID 1 in the container.
  • The process ID 1 in the container is the java process running tomcat.*)
  • Yes, tomcat itself shuts down gracefully, but not do so the servlets.
  • Servlets get killed after 2 seconds, even if they are in the middle of processing a reguest(!!)

*) Side note: Although our container entrypoint is [ "/opt/tomcat/bin/catalina.sh", "run" ], but in catalina.sh the java process is started via the bash buildin "exec" command, and therefore the java process replaces the shell process and hereby becomes the new process id 1. (I can verify this by exec into the running container and do a "ps aux" in there.) Btw, I am using tomcat 7.0.88.

I found statements about tomcat doing gracefull shutdown by default (http://tomcat.10.x6.nabble.com/Graceful-Shutdown-td5020523.html - "any in-progress connections will complete"), but all I can see is that the SIGTERM which is sent from docker to the java process results in hardly stopping the ongoing execution of a request.

I wrote a little rest servlet to test this behaviour:

import javax.ws.rs.*;
import javax.ws.rs.core.*;
import javax.ws.rs.core.Response.Status;

@Path("/")
public class SlowServerRes
{
    @GET
    @Produces(MediaType.TEXT_PLAIN)
    @Path("test1")
    public Response test1(@QueryParam("sleep") final int sleepDurationSec)
    {
        long received = System.currentTimeMillis();
        System.out.println("+++++++++++++++++++++ received request at " + received);

        for (int i=1; i <= sleepDurationSec; i++) {         
            System.out.println("  ++++ Sleeping for 1 sec ("+i+")");
            try { Thread.sleep(1000); }
            catch (InterruptedException e) { 
                System.out.println("   Sleep was interrupted at second " + i + " ... ignoring/continue sleeping."); 
            }
        }

        long finished = System.currentTimeMillis();
        String result = "received: " + received + "  finished: " + finished;
        System.out.println("+++++++++++++++++++++ " + result);
        Response response = Response.status(Status.OK).entity(result).build();
        return response;
    }
}

After intensive googling I finally came across this posting: http://grokbase.com/t/tomcat/users/113nayv5kx/tomcat-6-graceful-shutdown

So the grace period that is given to tomcat is NOT propagated as a grace period for the servlets. I wonder weather this makes much sense, but it looks like this is the fact. So the only way to give servlets the possibility to properly end their ongoing requests is to change the "unloadDelay" (https://tomcat.apache.org/tomcat-7.0-doc/config/context.html).

However, I did not find the right place in the tomcat config files to define a non-default unloadDelay. In case this matters, my main concern is about jersey servlets (org.glassfish.jersey.servlet.ServletContainer).

Or maybe there are other possibilities, which I don't see by now?

(I added kubernetes to the tags list, because this may be of major concern especially for kubernets, as it relocates (docker stop->SIGTERM) containers quite often, just to keep the load balanced.)

-- user2081279
docker
java
kubernetes
shutdown
tomcat

2 Answers

8/3/2018

You could expose a REST API to stop gracefully the servers.

1) Implement and use a javax filter that maintains in its own state the HTTP requests in progress.
2) As the stop event occurs, the current tomcat instance doesn't have to serve any longer new clients requests. So make sure that new requests cannot be redirected to this instance.
3) As the stop event occurs (second thing), start a thread that waits for all requests were served. As all responses were sent, request the tomcat instance to shutdown : the command String shutdown may be a way.

-- davidxxx
Source: StackOverflow

8/6/2018

Now I found the answer here: https://stackoverflow.com/a/11154770/2081279

It worked for me under linux with

<Context path="/myapp" unloadDelay="10000"/> 

but only with upper letter "C"ontext.

-- user2081279
Source: StackOverflow