I have Kubernetes cluster running on AWS instances, and prometheus running inside kubernetes for monitoring. There are three etcd servers running external to kubernetes, and I am trying to use prometheus to monitor etcd health.
Prometheus is deployed as a statefulset, and has metrics for kubelet, node-exporters, and itself. However, I cannot get any metrics from etcd.
Here is the relevant part of prometheus's config:
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus
namespace: monitoring
data:
prometheus.yml: |-
global:
scrape_interval: 30s
evaluation_interval: 30s
rule_files:
- /etc/alertmanager/*.rules
scrape_configs:
- job_name: etcd
scheme: https
static_configs:
- targets: ['x.x.x.x:2379']
tls_config:
ca_file: /etc/etcd/ssl/ca.pem
cert_file: /etc/etcd/ssl/client.pem
key_file: /etc/etcd/ssl/client-key.pem
insecure_skip_verify: true
- job_name: kubelets
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/tokenThis is the error I am getting in Prometheus's dashboard:
Get https://x.x.x.x.:2379/metrics: x509: cannot validate certificate for x.x.x.x because it doesn't contain any IP SANsThe cert is self-signed, but shouldn't "insecure_skip_verify" take care of that?
To eliminate etcd issues, if you're using etcd3, you can use the following arguments with the etcd client etcdctl and interact with the etcd server using the steps in https://github.com/coreos/etcd/blob/master/Documentation/dev-guide/interacting_v3.md. If it works without errors, I'd say this is a prometheus issue for not honoring the insecure_skip_verify: true configuration.
--insecure-skip-tls-verify=true skip server certificate verification
--insecure-transport=true disable transport security for client connections