I've got prometheus running ontop of kubernetes with the following scrape config, as described by the documentation. Where the .pem files are located on disk within the prometheus container.

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#

scrape_configs:
- job_name: etcd
  static_configs:
  - targets: ['10.0.0.222:2379','10.0.0.221:2379','10.0.0.220:2379']
  tls_config:
  # CA certificate to validate API server certificate with.
    ca_file: /prometheus/ca.pem
    cert_file: /prometheus/cert.pem
    key_file: /prometheus/key.pem

I see that etcd as a target in prometheus, however its returning garbage.

https://i.imgur.com/rdRI4V7.png

I am able to hit the metrics endpoint doing a local curl by passing in the client certificate information like so.

What am I doing wrong?

sudo curl --cacert /etc/ssl/etcd/ssl/ca.pem https://127.0.0.1:2379/metrics -L --cert /etc/ssl/etcd/ssl/node-kubemaster-rwva1-prod-2.pem --key /etc/ssl/etcd/ssl/node-kubemaster-rwva1-prod-2-key.pem^C