I am running a java application on tomcat in a k8s cluster, I am using a prestop lifecycle hook to run /usr/local/tomcat/bin/shutdown.sh
before the container gets terminated. Also defining a termination grace period of 60 secs.
expected behaviour: tomcat shutsdown gracefully when I kill a pod or delete a deployment.
actual behaviour: tomcat shutsdown abruptly.
Any thoughts? Thanks in advance.
apiVersion: apps/v1
kind: Deployment
metadata:
name: sample
annotations:
spec:
replicas: 2
selector:
matchLabels:
app: sample
name: sample
template:
metadata:
labels:
app: sample
name: sample
spec:
hostname: sample-web
terminationGracePeriodSeconds: 60
volumes:
- name: splunk-inputs
configMap:
name: splunk-conf
items:
- key: inputs.conf
path: ./inputs.conf
- name: splunk-outputs
configMap:
name: splunk-conf
items:
- key: outputs.conf
path: ./outputs.conf
- name: docker-socket
hostPath:
path: /var/run/docker.sock
- name: tomcat-log-common
emptyDir: {}
containers:
- name: sample
image: sampleregistery.azurecr.io/root
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command: ["/usr/local/tomcat/bin/shutdown.sh"]
found the issue:
I was running this prestop lifecycle hook:
lifecycle: preStop: exec: command: ["/usr/local/tomcat/bin/catalina.sh"]
It should be something like:
lifecycle: preStop: exec: command: ["/usr/local/tomcat/bin/catalina.sh && sleep 30"]
In the first case, K8s was attempting to exit out of the lifecycle hook as soon as it invokes the shutdown.sh without waiting for tomcat to drain all threads.
it just takes about 5-10 secs depending upon your app. giving a sleep of 30 secs will keep the prestop hook active which should give enough time for tomcat to finish the shutdown process" .