I am scraping metrics of a service on a Kubernetes cluster. I just annotated the service so that Prometheus detects and start scraping automatically. The following annotations were added to the service:
metadata:
annotations:
prometheus.io/path: /minio/prometheus/metrics
prometheus.io/port: "9000"
prometheus.io/scrape: "true"
This works fine as long as the service is not configured with TLS. However when I enable TLS (HTTPS) Prometheus is no longer able to discover and scrape the service automatically. To scrape from TLS enabled server I need to explicitly add the scrape endpoint with insecure_skip_verify. I am using self signed certificate.
Is there any way I can configure Prometheus in such way that it can discover and start scraping endpoints automatically when endpoints are TLS enabled with self signed certificate?
The simplest approach is for your app to expose a non-TLS metrics deicated port so you can have all traffic secured and metrics exposed with clear http. That is the way that's suggested for ie. Istio for mTLS and healthchecks
You can use a separate port for health check and enable mutual TLS only
on the regular service port. other then that you can add tls config to your scrape targets so if you have your own selfsigned CA add it's cert here and it should be fine.