K
Q

Custom alert for pod memory utilisation in Prometheus

7/26/2018

I created alert rules for pod memory utilisation, in Prometheus. Alerts are showing perfectly on my slack channel, but it do not contain the name of the pod so that difficult to understand which pod is having the issue .

It is Just showing [FIRING:35] (POD_MEMORY_HIGH_UTILIZATION default/k8s warning). But when I look in to the "Alert" section in the Prometheus UI, I can see the fired rules with its pod name. Can anyone help?

My alert notification template is as follows:

alertname: TargetDown
      alertname: POD_CPU_HIGH_UTILIZATION
      alertname:  POD_MEMORY_HIGH_UTILIZATION
receivers:

    - name: 'slack-notifications'

      slack_configs:

      - channel: '#devops'
        title: '{{ .CommonAnnotations.summary }}'
        text: '{{ .CommonAnnotations.description }}'

        send_resolved: true

I have added the option title: '{{ .CommonAnnotations.summary }}' text: '{{ .CommonAnnotations.description }}' in my alert notification template and now it is showing the description. My description is description: pod {{$labels.pod}} is using high memory. But only showing is using high memory. Not specifying the pod name

-- manu thankachan
kubernetes
prometheus-alertmanager

Similar Questions

kubectl apply command does not work, gives connection refused error
How I can use MongoDB GUI tool like mongo-express or RockMongo on Kubernetes cluster
Kubernetes: AKS unable to view the site
How to access K8s's flannel network from outside

1 Answer

7/27/2018

As mentioned in the article, you should check the alert rules and update them if necessary. See an example:

ALERT ElasticacheCPUUtilisation
  IF aws_elasticache_cpuutilization_average > 80
  FOR 10m
  LABELS { severity = "warning" }
    ANNOTATIONS {
    summary = "ElastiCache CPU Utilisation Alert",
    description = "Elasticache CPU Usage has breach the threshold set (80%) on cluster id {{ $labels.cache_cluster_id }}, now at {{ $value }}%",
    runbook = "https://mywiki.com/ElasticacheCPUUtilisation",
  }

To provide external URL for your prometheus GUI, apply CLI argument to your prometheus server and restart it:

-web.external-url=http://externally-available-url:9090/

After that, you can put the values into your alertmanager configuration. See an example:

receivers:
  - name: 'iw-team-slack'
    slack_configs:
    - channel: alert-events
      send_resolved: true
      api_url: https://hooks.slack.com/services/<your_token>
      title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] Monitoring Event Notification'
      text: >-
        {{ range .Alerts }}
           *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
          *Description:* {{ .Annotations.description }}
          *Graph:* <{{ .GeneratorURL }}|:chart_with_upwards_trend:> *Runbook:* <{{ .Annotations.runbook }}|:spiral_note_pad:>
          *Details:*
          {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
          {{ end }}
        {{ end }}
-- VAS
Source: StackOverflow