How to enable Alert manager pick alert manager configs from a new kubernetes namespace?

11/7/2021

Alertmanager(kind: Alertmanager) is running in prometh namespace.

We have access to push alert rule definitions(kind: PrometheusRule) to prometh namespace, with additional labels in alert payload(sourceNamespace & destinationNamespace), as shown below:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus: service-prometheus
    role: alert-rules
    app: kube-prometheus-stack
    release: monitoring-prom
  name: rule_a
  namespace: monitoring
spec:
  groups:
    - name: rule_a_alert
      rules:
        - alert: usage_exceed
          expr: salesforce_api_usage > 100000
          labels:
            severity: urgent
            sourceNamespace: app-dev
            destinationNamespace: prometh

Value of sourceNamespace indicate the namespace where application pods are running.

We have alert manager config(kind: AlertmanagerConfig) that match with rule definition(above) using below syntax:

route:
  routes:
  - match:
      severity: urgent

but we do not have access to push alert manager config into prometh namespace. we have access to push alert manager config only into app-dev


Is it possible to start alert manager, by enabling it to register new alertmanager configs from a new sourceNamespace in runtime? with alertmanager.alertmanagerSpec.alertmanagerConfigNamespaceSelector field

Related discussion: https://issueexplorer.com/issue/prometheus-operator/prometheus-operator/4263

-- overexchange
kubernetes
prometheus
prometheus-alertmanager
prometheus-operator

0 Answers