Kubernetes (K8s) - Get alert from Prometheus/Grafana when a new container is not created with a period of time

6/6/2019

I have a system in which a container is created every hour and I want to get an alert when a container is not created for a certain period of time from Prometheus or Grafana. Is there anyway to get a count of containers created in last let say X-hr and send an alert if the count of containers are less? The containers have a name template like : 2019-month-date-hour
Eg: 2019-5-11-23

I tried to use this query to get the count of containers created in a month count(count(container_last_seen{container_name=~"2019-5.*", namespace=$namespace}) by (container_name)) but not able to set grafana alert on this query for a period of time.

-- Aman Kr Singh
grafana-alerts
kubernetes
promql

1 Answer

6/6/2019

This is what you should do

  1. use event exporter to capture the events generated in kubernetes cluster
  2. get those events in promethues
  3. create prometheus rule to check for container/pod creation event
  4. get prometheus invoke alert via alert manager using the rule
-- P Ekambaram
Source: StackOverflow