Can it b possible to add nested query in prometheus alerts? I am using prometheus to monitor kubernetes cluster.
Alert is generated if node is down but i want to configure alert so that if any node is down then prometheus should not send alert for pods and services which are running on that.
Something like this,
if(pod_down)
if(corresponding_node_down)
//dont send alert
//node down alert is in firing stateIf a node is down, then Kubernetes should be handling that automatically and moving services elsewhere. Accordingly an alert like this isn't very useful.
What I'd suggest you do it alert on user-visible symptoms such as high latency and error ratios rather than individual causes such as a machine or container being down.
Take a look at this Prometheus Alertmanager configuration.
link: prometheus.io/docs/alerting/configuration/#inhibit_rule
Inhibition is a concept of suppressing notifications for certain alerts if certain other alerts are already firing.
Example: An alert is firing that informs that an entire cluster is not reachable. Alertmanager can be configured to mute all other alerts concerning this cluster if that particular alert is firing. This prevents notifications for hundreds or thousands of firing alerts that are unrelated to the actual issue.