If I know for a fact during startup that my service will not work.. Is there anything I can return from one of the probes that will tell Kubernetes to act as if the pod is no longer usable and needs to be restarted?
Right now as far as I understand the only responses are 200-399 which will be handled as "success" and anything else which is regarded as a failure. Are there any other options?
You should use the readinessProbe
in your deployments to check whether your pod is ready to serve the requests or not. There are three types of readinessProbes (HTTP, TCP, Exec) and the one which you need for infrastructure should be exec
, as you can run the command in the shell to test that. Follow the below e.g.:-
readinessProbe:
exec:
command:
- cat
- /tmp/healthy
initialDelaySeconds: 5
periodSeconds: 5
Similar way, you should configure the exec
livensesProbes as well.
livenessProbe:
exec:
command:
- cat
- /tmp/healthy
initialDelaySeconds: 5
periodSeconds: 5
If your process can determine that it's in an unsalvageable state, it doesn't need to wait for a probe to happen; it can just exit on its own, and Kubernetes will restart it. How exactly to do this depends on your language but usually there is a function called exit()
, or if your language has exceptions, throwing an exception and not catching it will generally terminate the program.
From a Kubernetes point of view this looks very similar to failing a liveness probe. The container will terminate and restart; if it does that more than a couple of times, the cluster will start delaying the restarts, with the CrashLoopBackOff
state. Leaving yourself a hint by writing a diagnostic message before you exit is helpful, so that kubectl logs
will explain why the process keeps exiting.
No, there are no other ways to detect that except 3 which are described in a document you linked:
Probably in your case you can use exec
check and something like a flag file.
For example, your app can create a file "/tmp/iamdead", and you can create a liveness probe like:
livenessProbe:
exec:
command:
- sh
- -c
- "cat /tmp/iamdead; if [ $? -eq 0 ]; then exit 1; fi"