Prometheus Alert for Multiple Deployments

3/31/2021

I need to implement an alert for a prometheus metric that is being exposed by many instances of a given application running on a kubernetes cluster.

The alert has to be created in a .yaml file in the following format:

- name: some-alert-name
  interval: 30s
  rules:
  - alert: name-alert
    expr: <Expression To Make>
    labels:
      event_id: XXXXX
    annotations:
      description: "Project {{ $labels.kubernetes_namespace }} / App {{ $labels.app }} / Pod {{ $labels.kubernetes_pod_name }} / Instance {{ $labels.instance }}."
      summary: "{{ $labels.kubernetes_namespace }}"

The condition to applied to the alert would be something like: givenMetricValue > 4

I have no issue in getting the metric values for all instances, as I can do it with: metricName{app=~"common-part-of-deployments-name-.*"}"

My troubles are in having a unique alert with an expression that fires if one of them satisfies the condition.

Is this possible to be done? If so, how can I do it?

-- LeYAUable
alert
kubernetes
monitoring
prometheus
prometheus-alertmanager

1 Answer

4/5/2021

Turns out, if you want create the alert with a generic "all-fetching" expression like

metricName{app=~"common-part-of-deployments-name-.*"}"

The alert will be triggered for each deployment that the regex matches. So all you need is an alert with a generic expression.

-- LeYAUable
Source: StackOverflow