How do I set up a hook to send an email on Kubernetes pod restart?

4/15/2019

When my Kubernetes pods get terminated (due to restarting or completely stopping), I would like to invoke some kind of a lifecycle hook that will notify me of the termination through email. Something like the following:

  onTermination:
    args:
    - '/bin/sh'
    - '-c'
    - | 
       <smtp login and send email script>

How I can get an email when my pod is restarted or stop in Kubernetes?

-- Raghunath Babalsure
docker
kubernetes

1 Answer

4/17/2019

Only way I know that you can get an email if there is something wrong with Cluster/Node/Pod are monitoring tools.

You can use i.e paid software with free trial like sysdig or datadog. If you want to learn Kubernetes metrics you can use kube-state-metrics with Prometheus (AlertManager) and i.e Grafana as backend.

Here you have some steps which may be useful.

  1. Install kube-state-metrics.
  2. Install Prometheus
  3. Install Grafana
  4. Connect to Prometheus (kubectl Port Forwarding or expose Prometheus as a Service)
  5. Connect to Grafana (kubectl port forwarding)
  6. In Grafana you have something like Alerts > Notification Chanels. There you can define how notification can be send (one of the option is email).
  7. Create dashboard with desired metrics and add alerting to it.

You can also check InfluxDB or Stackdriver as data source.

Tutorials which may help you
https://devopscube.com/setup-prometheus-monitoring-on-kubernetes/
https://itnext.io/kubernetes-monitoring-with-prometheus-in-15-minutes-8e54d1de2e13

-- PjoterS
Source: StackOverflow