Can a probe in kubernetes tell the infrastructure that the service has definitely failed?

5/24/2020

If I know for a fact during startup that my service will not work.. Is there anything I can return from one of the probes that will tell Kubernetes to act as if the pod is no longer usable and needs to be restarted?

Right now as far as I understand the only responses are 200-399 which will be handled as "success" and anything else which is regarded as a failure. Are there any other options?

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

-- Shahar Prish
docker
kubernetes

3 Answers

5/24/2020

You should use the readinessProbe in your deployments to check whether your pod is ready to serve the requests or not. There are three types of readinessProbes (HTTP, TCP, Exec) and the one which you need for infrastructure should be exec, as you can run the command in the shell to test that. Follow the below e.g.:-

readinessProbe:
  exec:
    command:
    - cat
    - /tmp/healthy
  initialDelaySeconds: 5
  periodSeconds: 5

Similar way, you should configure the exec livensesProbes as well.

livenessProbe:
  exec:
    command:
    - cat
    - /tmp/healthy
  initialDelaySeconds: 5
  periodSeconds: 5

Ref:- https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

-- nischay goyal
Source: StackOverflow

5/26/2020

If your process can determine that it's in an unsalvageable state, it doesn't need to wait for a probe to happen; it can just exit on its own, and Kubernetes will restart it. How exactly to do this depends on your language but usually there is a function called exit(), or if your language has exceptions, throwing an exception and not catching it will generally terminate the program.

From a Kubernetes point of view this looks very similar to failing a liveness probe. The container will terminate and restart; if it does that more than a couple of times, the cluster will start delaying the restarts, with the CrashLoopBackOff state. Leaving yourself a hint by writing a diagnostic message before you exit is helpful, so that kubectl logs will explain why the process keeps exiting.

-- David Maze
Source: StackOverflow

5/24/2020

No, there are no other ways to detect that except 3 which are described in a document you linked:

  1. HTTP check which detects response code.
  2. TCP probe which check is the port open or not.
  3. Exec which runs script inside your container and check its exit code.

Probably in your case you can use exec check and something like a flag file.

For example, your app can create a file "/tmp/iamdead", and you can create a liveness probe like:

livenessProbe:
  exec:
    command:
    - sh
    - -c
    - "cat /tmp/iamdead; if [ $? -eq 0 ]; then exit 1; fi"
-- Anton Kostenko
Source: StackOverflow