Im using prometheus operator 0.3.4 and alert manager 0.20 and it doesnt work, i.e. I see that the alert is fired (on prometheus UI on the alerts tab) but I didnt get any alert to the email. by looking at the logs I see the following , any idea ? please see the warn in bold maybe this is the reason but not sure how to fix it...

This is the helm of prometheus operator which I use: https://github.com/helm/charts/tree/master/stable/prometheus-operator

level=info ts=2019-12-23T15:42:28.039Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)"
level=info ts=2019-12-23T15:42:28.039Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)"
level=warn ts=2019-12-23T15:42:28.109Z caller=cluster.go:228 component=cluster msg="failed to join cluster" err="1 error occurred:\n\t* Failed to resolve alertmanager-monitoring-prometheus-oper-alertmanager-0.alertmanager-operated.monitoring.svc:9094: lookup alertmanager-monitoring-prometheus-oper-alertmanager-0.alertmanager-operated.monitoring.svc on 100.64.0.10:53: no such host\n\n"
level=info ts=2019-12-23T15:42:28.109Z caller=cluster.go:230 component=cluster msg="will retry joining cluster every 10s"
level=warn ts=2019-12-23T15:42:28.109Z caller=main.go:322 msg="unable to join gossip mesh" err="1 error occurred:\n\t* Failed to resolve alertmanager-monitoring-prometheus-oper-alertmanager-0.alertmanager-operated.monitoring.svc:9094: lookup alertmanager-monitoring-prometheus-oper-alertmanager-0.alertmanager-operated.monitoring.svc on 100.64.0.10:53: no such host\n\n"
level=info ts=2019-12-23T15:42:28.109Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s
level=info ts=2019-12-23T15:42:28.131Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/config/alertmanager.yaml
level=info ts=2019-12-23T15:42:28.132Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/config/alertmanager.yaml
level=info ts=2019-12-23T15:42:28.134Z caller=main.go:416 component=configuration msg="skipping creation of receiver not referenced by any route" receiver=AlertMail
level=info ts=2019-12-23T15:42:28.134Z caller=main.go:416 component=configuration msg="skipping creation of receiver not referenced by any route" receiver=AlertMail2
level=info ts=2019-12-23T15:42:28.135Z caller=main.go:497 msg=Listening address=:9093
level=info ts=2019-12-23T15:42:30.110Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00011151s
level=info ts=2019-12-23T15:42:38.110Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.000659096s

this is my config yaml

global:
  imagePullSecrets: []


prometheus-operator:
  defaultRules:
  grafana:
    enabled: true
  prometheusOperator:
    tolerations:
      - key: "WorkGroup"
        operator: "Equal"
        value: "operator"
        effect: "NoSchedule"
      - key: "WorkGroup"
        operator: "Equal"
        value: "operator"
        effect: "NoExecute"
    tlsProxy:
      image:
        repository: squareup/ghostunnel
        tag: v1.4.1
        pullPolicy: IfNotPresent
    resources:
      limits:
        cpu: 8000m
        memory: 2000Mi
      requests:
        cpu: 2000m
        memory: 2000Mi
    admissionWebhooks:
      patch:
        priorityClassName: "operator-critical"
        image:
          repository: jettech/kube-webhook-certgen
          tag: v1.0.0
          pullPolicy: IfNotPresent
    serviceAccount:
      name: prometheus-operator
    image:
      repository: quay.io/coreos/prometheus-operator
      tag: v0.34.0
      pullPolicy: IfNotPresent

  prometheus:
    prometheusSpec:
      replicas: 1
      serviceMonitorSelector:
        role: observeable
      tolerations:
        - key: "WorkGroup"
          operator: "Equal"
          value: "operator"
          effect: "NoSchedule"
        - key: "WorkGroup"
          operator: "Equal"
          value: "operator"
          effect: "NoExecute"
      ruleSelector:
        matchLabels:
          role: alert-rules
          prometheus: prometheus
      image:
        repository: quay.io/prometheus/prometheus
        tag: v2.13.1
  alertmanager:
    alertmanagerSpec:
      image:
        repository: quay.io/prometheus/alertmanager
        tag: v0.20.0
      resources:
        limits:
          cpu: 500m
          memory: 1000Mi
        requests:
          cpu: 500m
          memory: 1000Mi
    serviceAccount:
      name: prometheus
    config:
      global:
        resolve_timeout: 1m
        smtp_smarthost: 'smtp.gmail.com:587'
        smtp_from: 'alertmanager@vsx.com'
        smtp_auth_username: 'ds.monitoring.grafana@gmail.com'
        smtp_auth_password: 'mypass'
        smtp_require_tls: false
      route:
        group_by: ['alertname', 'cluster']
        group_wait: 45s
        group_interval: 5m
        repeat_interval: 1h
        receiver: default-receiver
        routes:
          - receiver: str
            match_re:
              cluster: "canary|canary2"

      receivers:
        - name: default-receiver
        - name: str
          email_configs:
          - to: 'rayndoll007@gmail.com'
            from: alertmanager@vsx.com
            smarthost: smtp.gmail.com:587
            auth_identity: ds.monitoring.grafana@gmail.com
            auth_username: ds.monitoring.grafana@gmail.com
            auth_password: mypass

        - name: 'AlertMail'
          email_configs:
            - to: 'rayndoll007@gmail.com'

https://codebeautify.org/yaml-validator/cb6a2781

The error says it failed in the resolve , the pod name called alertmanager-monitoring-prometheus-oper-alertmanager-0 which is up and running however it try to resolve : lookup alertmanager-monitoring-prometheus-oper-alertmanager-0.alertmanager-operated.monitoring.svc not sure why...

Here is the output of kubectl get svc -n mon

update This is warn logs

level=warn ts=2019-12-24T12:10:21.293Z caller=cluster.go:438 component=cluster msg=refresh result=failure addr=alertmanager-monitoring-prometheus-oper-alertmanager-0.alertmanager-operated.monitoring.svc:9094
level=warn ts=2019-12-24T12:10:21.323Z caller=cluster.go:438 component=cluster msg=refresh result=failure addr=alertmanager-monitoring-prometheus-oper-alertmanager-1.alertmanager-operated.monitoring.svc:9094
level=warn ts=2019-12-24T12:10:21.326Z caller=cluster.go:438 component=cluster msg=refresh result=failure addr=alertmanager-monitoring-prometheus-oper-alertmanager-2.alertmanager-operated.monitoring.svc:9094

This is the kubectl get svc -n mon

alertmanager-operated                     ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP   6m4s
monitoring-grafana                        ClusterIP   100.11.215.226   <none>        80/TCP                       6m13s
monitoring-kube-state-metrics             ClusterIP   100.22.248.232   <none>        8080/TCP                     6m13s
monitoring-prometheus-node-exporter       ClusterIP   100.33.130.77    <none>        9100/TCP                     6m13s
monitoring-prometheus-oper-alertmanager   ClusterIP   100.33.228.217   <none>        9093/TCP                     6m13s
monitoring-prometheus-oper-operator       ClusterIP   100.21.229.204   <none>        8080/TCP,443/TCP             6m13s
monitoring-prometheus-oper-prometheus     ClusterIP   100.22.93.151    <none>        9090/TCP                     6m13s
prometheus-operated                       ClusterIP   None             <none>        9090/TCP                     5m54s