I'm looking for a approach where we can turn on/off liveness probe using configMap UI on Rancher, and same time it does not require redeployment of the pod.
In details : Assume the liveness probe is already been config on pod. Next I have a boolean flag and it is set false, so if false, liveness probe will stop probing on pod. At other time, if set boolean flag to true, liveness probe will resume probe. All these need to works without pod redeployment. Below is my draft idea:
1) There's a configMap UI (configMap listed on Rancher) which hold the boolean flag to turn on/off liveness probe something like this :
app.liveness.probe.mode=false2) Next I want to absorb the above boolean flag into deployment.yaml with the conditional checking as below :
{{ if app.liveness.probe.mode }}
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 70
periodSeconds: 10
{{ end }}3) I'm not sure how to reference the configMap boolean flag on Rancher into the deployment.yaml file. 4) Or is there any other approach to control the liveness probe on/off toggle without redeployment of pod.
Sounds to me as if you want to temporarily switch off you Liveness probe in order to keep your Pod running, even if the Pod is technically not healthy.
Have you considered using a Readiness probe for this?
So, you remove your Liveness probe, which basically keeps your Pod running as long as your application is running.
Your define a Readiness probe that executes HTTP checks on an application endpoint. While that endpoint returns 200, your Pod receives traffic. If you want to take traffic off the Pod, while keeping your Pod running, your readiness probe will have to return an error code (>=400).
To accomplish this dynamically with ConfigMaps, you could mount a ConfigMap as a volume into your application container putting a config file onto your container file system. The config file contains a flag like readninessProbe.active=true. This file is read by your Readiness probe endpoint and returns a 200 or 500 dependent on the value of the boolean.
Naturally, a similar approach could be used to control your Livness probe endpoint, but the K8s mechanism for keeping Pods running but taking traffic off them is taken care of by Readiness probes.