Prometheus alertmanager slack notification newlines issue

10/9/2021

I've defined an alert for my kubernetes pods as described below to notify through slack. I used the example described in the official documentation for ranging over all received alerts to loop over multiple alerts and render them on my slack channel I do get notifications but the new lines do not get rendered correctly somehow. I'm new to prometheus any help is greatly appreciated. Thanks.

detection:
  # Alert If:
  # 1. Pod is not in a running state.
  # 2. Container is killed because it's out of memory.
  # 3. Container is evicted.
  rules:
    groups:
      - name: not-running
        rules:
          - alert: PodNotRunning
            expr: kube_pod_status_phase{phase!="Running"} > 0
            for: 0m
            labels:
              severity: warning
            annotations:
              summary: "Pod {{ $labels.pod }} is not running."
              description: 'Kubernetes pod {{ $labels.pod }} is not running.'
          - alert: KubernetesContainerOOMKilledOrEvicted
            expr: kube_pod_container_status_last_terminated_reason{reason=~"OOMKilled|Evicted"} > 0
            for: 0m
            labels:
              severity: warning
            annotations:
              summary: "kubernetes container killed/evicted (instance {{ $labels.instance }})"
              description: "Container {{ $labels.container }} in pod {{ $labels.namespace }}/{{ $labels.pod }}
                            has been OOMKilled/Evicted."

  route:
    group_by: ['alertname']
    group_wait: 30s
    group_interval: 3m
    repeat_interval: 4h
    receiver: slack-channel
    routes:
    - match:
        alertname: PodNotRunning
    - match:
        alertname: KubernetesContainerOOMKilledOrEvicted

notifications:
  receivers:
    - name: slack-channel
      slack_configs:
        - channel: kube-alerts
          title: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}"
          text: "{{ range .Alerts }}{{ .Annotations.description }}\n{{ end }}"

How it gets rendered on the actual slack channel:

Title: inst-1 down.\ninst-2 down.\ninst-3 down.\ninst-4 down.
Text: inst-1 down.\ninst-2 down.\ninst-3 down.\ninst-4 down

How I though it would render:

Title: inst-1 down.
Text: inst-1 down.

Title: inst-2 down.
Text: inst-2 down.

Title: inst-3 down.
Text: inst-3 down.


Title: inst-4 down.
Text: inst-4 down.
-- Pannu
kubernetes
prometheus-alertmanager
slack

0 Answers