sync all firing alerts to webhook at specific point in time from alertmanager

2/23/2022

Trying to resend all firing alarms to webhook at a specific point in time due to webhook requirement for synchronisation. Prometheus and alertmanager are deployed within a kubernetes cluster in gcp.

Any idea how to do this. The repeat_intervall is only resending one specific alert after 12h but need to send all of them at once.

find attached my alertmanager config

  config:
    global:
      resolve_timeout: 5m
    route:
      group_by: ['job']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      receiver: 'null'
      routes:
      - match:
          alertname: Watchdog
        receiver: 'null'
      - match_re:
          severity: ".*"
        receiver: 'webhook-prod1'
        continue: true
      - match_re:
          severity: ".*"
        receiver: 'webhook-prod2'
    receivers:
    - name: 'null'
    - name: 'webhook-prod1'
      webhook_configs:
      - send_resolved: true
        url: 'http://xx.xx.xx.xx:9090/probe/webhook/prometheus/'
        http_config:
          basic_auth:
            username: "username"
            password: <secret>
    - name: 'webhook-prod2'
      webhook_configs:
      - send_resolved: true
        url: 'http://xx.xx.xx.xx:9090/probe/webhook/prometheus/'
        http_config:
          basic_auth:
            username: "username"
            password: <secret>
    templates:
    - '/etc/alertmanager/config/*.tmpl'
-- enceladus2022
alertmanager
google-cloud-platform
kubernetes
prometheus
webhooks

1 Answer

2/23/2022

Looks like there's no such configuration in alertmanager for now.

Instead you could call the alertmanager api to fetch all currently firing alerts at a specific point in time in your webhook system, the same way as amtool does.

The swagger openapi docs is as follows:

Alertmanager API

-- YwH
Source: StackOverflow