I've defined an alert for my kubernetes pods as described below to notify through slack. I used the example described in the official documentation for ranging over all received alerts to loop over multiple alerts and render them on my slack channel I do get notifications but the new lines do not get rendered correctly somehow. I'm new to prometheus any help is greatly appreciated. Thanks.
detection:
# Alert If:
# 1. Pod is not in a running state.
# 2. Container is killed because it's out of memory.
# 3. Container is evicted.
rules:
groups:
- name: not-running
rules:
- alert: PodNotRunning
expr: kube_pod_status_phase{phase!="Running"} > 0
for: 0m
labels:
severity: warning
annotations:
summary: "Pod {{ $labels.pod }} is not running."
description: 'Kubernetes pod {{ $labels.pod }} is not running.'
- alert: KubernetesContainerOOMKilledOrEvicted
expr: kube_pod_container_status_last_terminated_reason{reason=~"OOMKilled|Evicted"} > 0
for: 0m
labels:
severity: warning
annotations:
summary: "kubernetes container killed/evicted (instance {{ $labels.instance }})"
description: "Container {{ $labels.container }} in pod {{ $labels.namespace }}/{{ $labels.pod }}
has been OOMKilled/Evicted."
route:
group_by: ['alertname']
group_wait: 30s
group_interval: 3m
repeat_interval: 4h
receiver: slack-channel
routes:
- match:
alertname: PodNotRunning
- match:
alertname: KubernetesContainerOOMKilledOrEvicted
notifications:
receivers:
- name: slack-channel
slack_configs:
- channel: kube-alerts
title: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}"
text: "{{ range .Alerts }}{{ .Annotations.description }}\n{{ end }}"
How it gets rendered on the actual slack channel:
Title: inst-1 down.\ninst-2 down.\ninst-3 down.\ninst-4 down.
Text: inst-1 down.\ninst-2 down.\ninst-3 down.\ninst-4 down
How I though it would render:
Title: inst-1 down.
Text: inst-1 down.
Title: inst-2 down.
Text: inst-2 down.
Title: inst-3 down.
Text: inst-3 down.
Title: inst-4 down.
Text: inst-4 down.