I need to implement an alert for a prometheus metric that is being exposed by many instances of a given application running on a kubernetes cluster.
The alert has to be created in a .yaml file in the following format:
- name: some-alert-name
interval: 30s
rules:
- alert: name-alert
expr: <Expression To Make>
labels:
event_id: XXXXX
annotations:
description: "Project {{ $labels.kubernetes_namespace }} / App {{ $labels.app }} / Pod {{ $labels.kubernetes_pod_name }} / Instance {{ $labels.instance }}."
summary: "{{ $labels.kubernetes_namespace }}"
The condition to applied to the alert would be something like: givenMetricValue > 4
I have no issue in getting the metric values for all instances, as I can do it with: metricName{app=~"common-part-of-deployments-name-.*"}"
My troubles are in having a unique alert with an expression that fires if one of them satisfies the condition.
Is this possible to be done? If so, how can I do it?
Turns out, if you want create the alert with a generic "all-fetching" expression like
metricName{app=~"common-part-of-deployments-name-.*"}"
The alert will be triggered for each deployment that the regex matches. So all you need is an alert with a generic expression.