Kubernetes Camel Spring applciation graceful shut down

10/31/2018

I have a spring boot camel application that processes messages on activeMq. however every once in a while, the ready probe fails and restarts the pod. Now thats fine, but then i get the applciation loggin out Inflight messages constantly and it gets into a restart cyclem due to ready probe constantly failing.

Is it possible to have kubernets allow the spring/camel app to gracefully shut down or consume any messages on the queue?

-- user1555190
apache-camel
kubernetes
spring-boot

1 Answer

10/31/2018

Using Container hooks, you can setup prestop that can look as follows:

apiVersion: v1
kind: Pod
metadata:
  name: lifecycle-demo
spec:
  containers:
  - name: lifecycle-demo-container
    image: nginx
    lifecycle:
      postStart:
        exec:
          command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
      preStop:
        exec:
          command: ["/usr/sbin/nginx","-s","quit"]

This hook is called immediately before a container is terminated. It is blocking, meaning it is synchronous, so it must complete before the call to delete the container can be sent. No parameters are passed to the handler.

You can follow the flow on Termination of Pods

  1. User sends command to delete Pod, with default grace period (30s)
  2. The Pod in the API server is updated with the time beyond which the Pod is considered “dead” along with the grace period.
  3. Pod shows up as “Terminating” when listed in client commands
  4. (simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating because the time in 2 has been set, it begins the pod shutdown process.
    1. If the pod has defined a preStop hook, it is invoked inside of the pod. If the preStop hook is still running after the grace period expires, step 2 is then invoked with a small (2 second) extended grace period.
    2. The processes in the Pod are sent the TERM signal.
  5. (simultaneous with 3) Pod is removed from endpoints list for service, and are no longer considered part of the set of running pods for replication controllers. Pods that shutdown slowly cannot continue to serve traffic as load balancers (like the service proxy) remove them from their rotations.
  6. When the grace period expires, any processes still running in the Pod are killed with SIGKILL.
  7. The Kubelet will finish deleting the Pod on the API server by setting grace period 0 (immediate deletion). The Pod disappears from the API and is no longer visible from the client.
-- Crou
Source: StackOverflow