I'm running a Kubernetes cluster on Google cloud. My cluster has a deployment that exposing the health-check interface (over HTTP). In my deployment yaml file, I configured:
livenessProbe:
# an http probe
httpGet:
path: /hc
port: 80
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 60
periodSeconds: 90If my health check endpoint return anything but 200, the pod will be killed and restarted.
Currently, after pod restart, it just counts it on the "restart" counter but not notify anyone. I want to notify the sysadmin that this event has happened. I thought to notify with a webhook.
Is this possible? If not, what is my other notification alternative?
The somewhat convoluted standard answer to this is Kubernetes -> kube-state-metrics -> Prometheus -> alertmanager -> webhook. This might sound like a lot for a simple task, but Prometheus and its related tools are used much more broadly for metrics and alerting. If you wanted a more narrow answer, you could check out Brigade perhaps? But probably just use kube-prometheus (which is Prom with a bunch of related components all setup for you).
You can add a preStop hook to your pod spec. The hook can either run a script or make an HTTP call before the pod shuts down. You can configure the hook to call an API which triggers a notification.