Trying to resend all firing alarms to webhook at a specific point in time due to webhook requirement for synchronisation. Prometheus and alertmanager are deployed within a kubernetes cluster in gcp.
Any idea how to do this. The repeat_intervall is only resending one specific alert after 12h but need to send all of them at once.
find attached my alertmanager config
config:
global:
resolve_timeout: 5m
route:
group_by: ['job']
group_wait: 30s
group_interval: 5m
repeat_interval: 12h
receiver: 'null'
routes:
- match:
alertname: Watchdog
receiver: 'null'
- match_re:
severity: ".*"
receiver: 'webhook-prod1'
continue: true
- match_re:
severity: ".*"
receiver: 'webhook-prod2'
receivers:
- name: 'null'
- name: 'webhook-prod1'
webhook_configs:
- send_resolved: true
url: 'http://xx.xx.xx.xx:9090/probe/webhook/prometheus/'
http_config:
basic_auth:
username: "username"
password: <secret>
- name: 'webhook-prod2'
webhook_configs:
- send_resolved: true
url: 'http://xx.xx.xx.xx:9090/probe/webhook/prometheus/'
http_config:
basic_auth:
username: "username"
password: <secret>
templates:
- '/etc/alertmanager/config/*.tmpl'
Looks like there's no such configuration in alertmanager for now.
Instead you could call the alertmanager api to fetch all currently firing alerts at a specific point in time in your webhook system, the same way as amtool
does.
The swagger openapi docs is as follows: