We have a smaller issue when restarting pods when other pods connect there through a service (with ClusterIP). When the pod goes down there is a small timespan (~200ms) where some client requests fail, because they still try to connect to the pod which just went down. This results in some "Connection Refused" errors. We can avoid it by adding a preStop hook which sleeps a second. Unfortunately, it's ugly to do this for every deployment.
Is this maybe a known issue? Any idea how to solve this in general?
To me it looks like some kind of race condition (aka the pod is gone before it's properly deregistered from its endpoint)
We're using Kubernetes v1.5.3.
This is what the workaround looks like (simplified):
apiVersion: apps/v1beta1
kind: StatefulSet
spec:
template:
spec:
containers:
- name: solr
lifecycle:
preStop:
exec:
command:
- sleep
- "1"The example in the docs actually shows the issue. It begins to shutdown the process and removing the endpoint at the same time. So it probably already shutdown the before the endpoint is registered.
I found a post saying a common pitfall that could be your case:
Let’s say your Dockerfile ends with a CMD in the shell form:
CMD myapp
The shell form runs the command with /bin/sh -c myapp, so the process that will get the SIGTERM is actually /bin/sh and not its child myapp. Depending on the actual shell you’re running, it could or could not pass the signal to its children.
It probably being your app was killed violently causing the ~200ms connection fail. Maybe try the options suggest in the article?