Alertmanager(kind: Alertmanager
) is running in prometh
namespace.
We have access to push alert rule definitions(kind: PrometheusRule
) to prometh
namespace, with additional labels in alert payload(sourceNamespace
& destinationNamespace
), as shown below:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
labels:
prometheus: service-prometheus
role: alert-rules
app: kube-prometheus-stack
release: monitoring-prom
name: rule_a
namespace: monitoring
spec:
groups:
- name: rule_a_alert
rules:
- alert: usage_exceed
expr: salesforce_api_usage > 100000
labels:
severity: urgent
sourceNamespace: app-dev
destinationNamespace: prometh
Value of sourceNamespace
indicate the namespace where application pods are running.
We have alert manager config(kind: AlertmanagerConfig
) that match with rule definition(above) using below syntax:
route:
routes:
- match:
severity: urgent
but we do not have access to push alert manager config into prometh
namespace. we have access to push alert manager config only into app-dev
Is it possible to start alert manager, by enabling it to register new alertmanager configs from a new sourceNamespace
in runtime? with alertmanager.alertmanagerSpec.alertmanagerConfigNamespaceSelector
field
Related discussion: https://issueexplorer.com/issue/prometheus-operator/prometheus-operator/4263