Error in scraping node exporter from Prometheus running on Kubernetes

7/20/2017

I am running a Prometheus pod on Kubernetes cluster. I have a node-exporter installed on an instance in Openstack. It is running fine. I added its configuration in prometheus config file. After reloading, the target node-exporter shows up but its status is Down and the error is context deadline exceeded. I tried increasing thescrape-timeout but it also dint work. Rest all targets are working fine and are UP. I tried to ping the target endpoint from inside container and it is also fine. But unfortunately, there is no curl or wget inside container to check the metrics endpoint. This is the config file snippet.

- job_name: "node-exporter"
  metrics_path: '/metrics/'
  static_configs:
    - targets: ['192.168.248.243:9100']

I checked the prometheus logs and here they are.

time="2017-07-20T12:53:37Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:549" 
time="2017-07-20T12:53:44Z" level=info msg="Done checkpointing in-memory metrics and chunks in 6.631738129s." source="persistence.go:573" 
time="2017-07-20T12:54:18Z" level=error msg="Watch nodes unexpectedly closed: EOF" source="node.go:162" 
time="2017-07-20T12:58:44Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:549" 
time="2017-07-20T12:58:50Z" level=info msg="Done checkpointing in-memory metrics and chunks in 6.154427033s." source="persistence.go:573" 
time="2017-07-20T12:58:58Z" level=error msg="Watch pods unexpectedly closed: EOF" source="pod.go:169" 
time="2017-07-20T13:01:13Z" level=info msg="Loading configuration file /etc/prometheus/conf/prometheus.yml" source="main.go:247" 
time="2017-07-20T13:01:42Z" level=error msg="Watch pods unexpectedly closed: EOF" source="pod.go:169" 

My Kubernetes version is 1.6.4.

-- Shubham Kaushik
kubernetes
prometheus

0 Answers