helm prometheus operator - set email notifications/edit secrets

3/23/2020

I have installed prometheus-operator via helm and now want to set email notifications for alert manager:

I edited prometheus operator secret

    kubectl get secret alertmanager-prometheus-prometheus-oper-alertmanager -n monitoring -o yaml
    apiVersion: v1
    data:
      alertmanager.yaml: Z2xvYmFsOgogIHJlc29sdmVfdGltZW91dDogNW0KcmVjZWl2ZXJzOgotIG5hbWU6ICJudWxsIgpyb3V0ZToKICBncm91cF9ieToKICAtIGpvYgogIGdyb3VwX2ludGVydmFsOiA1bQogIGdyb3VwX3dhaXQ6IDMwcwogIHJlY2VpdmVyOiAibnVsbCIKICByZXBlYXRfaW50ZXJ2YWw6IDEyaAogIHJvdXRlczoKICAtIG1hdGNoOgogICAgICBhbGVydG5hbWU6IFdhdGNoZG9nCiAgICByZWNlaXZlcjogIm51bGwi
    kind: Secret
    metadata:
      creationTimestamp: "2020-03-23T09:19:57Z"
      labels:
        app: prometheus-operator-alertmanager
        chart: prometheus-operator-8.12.2
        heritage: Helm
        release: prometheus
      name: alertmanager-prometheus-prometheus-oper-alertmanager
      namespace: monitoring
      resourceVersion: "1853097"
      selfLink: /api/v1/namespaces/monitoring/secrets/alertmanager-prometheus-prometheus-oper-alertmanager
      uid: eb7f514e-6bf4-4791-9c1a-e45590ba2a36
    type: Opaque



 echo 'Z2xvYmFsOgogIHJlc29sdmVfdGltZW91dDogNW0KcmVjZWl2ZXJzOgotIG5hbWU6ICJudWxsIgpyb3V0ZToKICBncm91cF9ieToKICAtIGpvYgogIGdyb3VwX2ludGVydmFsOiA1bQogIGdyb3VwX3dhaXQ6IDMwcwogIHJlY2VpdmVyOiAibnVsbCIKICByZXBlYXRfaW50ZXJ2YWw6IDEyaAogIHJvdXRlczoKICAtIG1hdGNoOgogICAgICBhbGVydG5hbWU6IFdhdGNoZG9nCiAgICByZWNlaXZlcjogIm51bGwi' | base64 --decode

created new alertmanager.yaml file

global:
  resolve_timeout: 5m
route:
  group_by: [Alertname]
  # Send all notifications to me.
  receiver: email-alert

  group_by: ['job', 'alertname', 'service', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  receiver: email-alert
  routes:
  - match:
      severity: critical
    receiver: email-alert

receivers:
- name: email-alert
  email_configs:
  - to: email@example.com
    from: email@example.com
    # Your smtp server address
    smarthost: smtp.office365.com:587
    auth_username: email@example.com
    auth_identity: email@example.com
    auth_password: Pass

Created template:

apiVersion: v1
data:
  alertmanager.yaml: ALERTMANAGER_CONFIG
kind: Secret
metadata:
  name: alertmanager-main
  namespace: monitoring
type: Opaque

Encoded it and applied

sed "s/ALERTMANAGER_CONFIG/$(cat alertmanager.yaml | base64 -w0)/g" alertmanager-secret-k8s.yaml | kubectl apply -f -

I exposed alertmanager as Nodeport via port 30700

So when i access to it (http://IP:30700/#/status), i see this alertmanager.yaml is not applied, ie, secret is not changed.

What needs to be done so i can edit this prometheus alertmanager secret ?

tried with

helm upgrade prometheus stable/prometheus-operator --namespace monitoring -f alertmanager.yaml 

without help

enter image description here

-- overflowed
kubernetes
kubernetes-helm
prometheus-alertmanager
prometheus-operator

1 Answer

3/24/2020

Figured it out thanks to this reference

  • delete current secret:

kubectl delete secret alertmanager-prometheus-prometheus-oper-alertmanager -n monitoring

  • create file alertmanager.yaml:

    global:
      resolve_timeout: 5m
    route:
      receiver: 'email-alert'
      group_by: ['job']
    
    
      routes:
      - receiver: 'email-alert'
        match:
          alertname: etcdInsufficientMembers
        group_wait: 30s
        group_interval: 5m
        repeat_interval: 12h  
    
    receivers:
    - name: email-alert
      email_configs:
      - to: receiver@example.com
        from: sender@example.com
        # Your smtp server address
        smarthost: smtp.office365.com:587
        auth_username: sender@example.com
        auth_identity: sender@example.com
        auth_password: pass
    • create new secret with same name as old one:

kubectl create secret generic alertmanager-prometheus- prometheus-oper-alertmanager -n monitoring --from- file=alertmanager.yaml

-- overflowed
Source: StackOverflow