K
Q

Gracefully draining sessions from Dropwizard application using Kubernetes

4/19/2016

I have a Dropwizard application that holds short-lived sessions (phone calls) in memory. There are good reasons for this, and we are not changing this model in the near term.

We will be setting up Kubernetes soon and I'm wondering what the best approach would be for handling shutdowns / rolling updates. The process will need to look like this:

  1. Remove the DW node from the load balancer so no new sessions can be started on this node.
  2. Wait for all remaining sessions to be completed. This should take a few minutes.
  3. Terminate the process / container.

It looks like kubernetes can handle this if I make step 2 a "preStop hook" http://kubernetes.io/docs/user-guide/pods/#termination-of-pods

My question is, what will the preStop hook actually look like? Should I set up a DW "task" (http://www.dropwizard.io/0.9.2/docs/manual/core.html#tasks) that will just wait until all sessions are completed and CURL it from kubernetes? Should I put a bash script that polls some sessionCount service until none are left in the docker container with the DW app and execute that?

-- Jason B
dropwizard
kubernetes

Similar Questions

What's the recommended way to replace a bad GKE node instance?
External IP column is empty in the kubectl get services result set
kibana in the kebernets cluster doesn't work
Can't run a docker container on kubernetes in the interactive mode

1 Answer

4/20/2016

Assume you don't use the preStop hook, and a pod deletion request has been issued.

  1. API server processes the deletion request and modify the pod object.
  2. Endpoint controller observes the change and removes the pod from the list of endpoints.
  3. On the node, a SIGTERM signal is sent to your container/process.
  4. Your process should trap the signal and drains all existing requests. Note that this step should not take more than the defined TerminationGracePeriod defined in your pod spec.

Alternatively, you can use the preStop hook which blocks until all the requests are drained. Most likely, you'll need a script to accomplish this.

-- Yu-Ju Hong
Source: StackOverflow