I'm trying to get my head around how to get prometheus https://hub.helm.sh/charts/stable/prometheus collect etcd stats. I understand I need to set tls for it, but have a hard time to find good way to do it without manual additional ansible steps. Is there the way I can get etcd certs on worker node and mount them to prometheus pod?
Following the Monitoring External Etcd Cluster With Prometheus Operator you can easily configure Prometheus to scrape metrics from ETCD.
We can do all of that by creating certs as kubernetes secrets and adding a tlsConfig to our service monitor. Let me walk you through the whole process.
The steps are:
1) Create etcd service
2) Create/attach endpoints
for etcd service
3) Create service monitor with appropriate tlsconfig. below example
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
k8s-app: etcd
name: etcd
namespace: kube-system
spec:
endpoints:
- interval: 30s
port: metrics
scheme: https
tlsConfig:
caFile: /etc/prometheus/secrets/kube-etcd-client-certs/etcd-client-ca.crt
certFile: /etc/prometheus/secrets/kube-etcd-client-certs/etcd-client.crt
keyFile: /etc/prometheus/secrets/kube-etcd-client-certs/etcd-client.key
serverName: etcd-cluster
jobLabel: k8s-app
selector:
matchLabels:
k8s-app: etcd
4) Create Etcd Client Certificates
5) Create Kubernetes Secrets along with previously created certificate and key for prometheus and etcd ca. This will allow prometheus to securely connect to etcd. Example:
kubectl -n monitoring create secret kube-etcd-client-certs --from-file=etcd-client-ca.crt=etcd-client.ca.crt --from-file=etcd-client.crt=etcd-client.crt --from-file=etcd-client.key=etcd-client.key
6) Update prometheus.yaml to include there names of the created secrets.
7) delploy etcd-service,servicemonitor and prometheus manifests to cluster
kubectl apply -f etcd-service.yaml
kubectl apply -f etcd-serviceMon.yaml
kubectl apply -f prometheus-prometheus.yaml
Enjoy