I am using Promethues to monitor my Kubernetes cluster. All my microservices can be accessed using my HA Proxy.

My base Promethues config is :

- job_name: 'kubernetes_pods'
  tls_config:
    insecure_skip_verify: true
  kubernetes_sd_configs:
  - api_server: http://172.29.219.102:8080
    role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_host_ip]
      target_label: __address__
      regex: (.*)
      replacement: 172.29.219.110:8080

Where 172.29.219.110:8080 is the IP & Port of my standalone HA Proxy.

The endpoint that I am trying to monitor using Prometheus is /auth/health.

When I do a simple curl command from anywhere, I see :

# curl http://172.29.219.110:8080/auth/health
{"status":"UP"}

But when Prometheus tries to do it, the logs indicate :

level=warn ts=2017-12-15T16:40:48.301741927Z caller=scrape.go:673 component="target manager" scrape_pool=kubernetes_pods target=http://172.29.219.110:8080/auth/health msg="append failed" err="no token found"

This endpoint is publicly exposed and requires no authentication what so ever. So why does Promethues say :