In k8s docs on lifecycles you can read:

PreStop - This hook is called immediately before a container is terminated ...

This means that when pod termination starts, this prestop hook is executed before SIGTERM is sent to the container.

At the same time when preStop hook is started, k8s is starting countdown timer to wait terminationGracePeriodSeconds number of seconds before sending SIGKILL to container.

Notice that in your case when your preStop sleeps 600s and terminationGracePeriodSeconds is also set to 600s may cause race condition.

Have a look at this piece of code in kubernetes source code:

select {
case <-time.After(time.Duration(gracePeriod) * time.Second):
	klog.V(2).Infof("preStop hook for container %q did not complete in %d seconds", containerID, gracePeriod)
case <-done:
	klog.V(3).Infof("preStop hook for container %q completed", containerID)
}

As you can see, kubelet waits for whatever happens first. But in your case both of these cases take 600s and they finish more or less in the same time and this may lead to race condition. So either preStop finishes successfully first and then countdown finishes killing the pod or countdown timer finishes first, sends SIGKILL to the container killing everything running inside, that also means killing preStop and then preStop finishes with error Warning FailedPreStopHook. Also notice that exited with 137 means that process got killed by k8s with SIGKILL (137 - 128 = 9, where 9 is SIGKILL number).

And the most important part is that your application didn't even know that it is about to get terminated.

OK, so what can you do? How to properly use preStop?

preStop should be used to either notify main process running in container that it is about to get terminated and it should start preparing for it or to e.g. inform other members of app cluster that this instance of the app will be terminated. It can also be used as you described (with sleep) to wait for a while so that changes in iptables have time to propagate. You may have not been aware of it but as soon as termination process starts, k8s is reconfiguring network so that new connections are not created. This is why sometimes small delay is used to give k8s time to propagate the changes and to allow the app to respond to ongoing requests before its terminated.

So the best thing you could do is making your application aware of termination process and make it react to SIGTERM signal gracefully. Using long sleep won't solve the problem you are facing.

Also, here is some information about springboot you may find usefull.

• By default Spring Boot used to not support graceful shutdown as described here. They introduced the fix for this in version 2.3.
• If you are using older version you can use Springboot-Graceful-Shutdown.
• Also check this article about graceful shutdown spring boot applications.