I am working with Prometheus on Kubernetes and trying to send out the alerts to slack. The problem is that the alert is firing but is not getting sent out to slack. I am using Prometheus 1.18.1 with Kubernetes 1.9.
For now I am just trying to send out the ‘DeadMansSwitch’ alert which is built-in and I can see in UI that it is firing. My alertmanager.yaml (under prometheus-operator/contrib/kube-prometheus/assets/alertmanager) looks like this:
global:
resolve_timeout: 5m
slack_api_url: 'https://hooks.slack.com/services/AAABBBCCC/DDDEEEFFF/GGGHHHIII’
route:
group_by: ['job']
group_wait: 1s
group_interval: 1s
repeat_interval: 1s
receiver: ‘slack’
routes:
- match:
alertname: DeadMansSwitch
receiver: ‘slack’
receivers:
- name: ‘slack’
slack_configs:
- channel: ‘#channel-name’The config on the AlertManager UI shows this:
global:
resolve_timeout: 5m
smtp_require_tls: true
pagerduty_url: https://events.pagerduty.com/v2/enqueue
hipchat_api_url: https://api.hipchat.com/
opsgenie_api_url: https://api.opsgenie.com/
wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/
victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/
route:
receiver: "null"
group_by:
- job
routes:
- receiver: "null"
match:
alertname: DeadMansSwitch
group_wait: 30s
group_interval: 5m
repeat_interval: 12h
receivers:
- name: "null"
templates: []Questions:
- The config on the AlertManager UI is different from what I have in the alertmanager.yaml file. Where is this config (in UI) coming from?
- Is setting receivers in alertmanager.yaml not enough to send out alerts? Am I missing anything here?
- Am I making changes in the wrong yaml file?
tl;dr = that yaml is so malformed that I'm stunned it parsed at all.
It's caused by two bits of invalid YAML: the indentation and the use of "smart apostrophes"; You have:
receivers:
- name: ‘slack’
slack_configs:but it should be:
receivers:
- name: 'slack'
slack_configs:otherwise in YAML it makes name into an object with a property of slack_configs rather than an anonymous object containing two properties name and slack_configs
Every place you have a "quoted" string of slack in your posted config, you have used the smart apostrophes which from YAML's point-of-view makes the string literal as "\u2018slack\u2019" instead of what you meant (it's a separate question why you quoted the string to begin with, since it's not any different from 5m or DeadMansSwitch).
You also have a normal ascii apostrophe leading the slack_api_url: but a smart apostrophe closing that string.
The infinitely handy remarshal project contains yaml2json which is great for spotting weird YAML-isms like that