I'm trying to get Heapster/InfluxDB/Grafana set up on a Kubernetes cluster. It looks like Heapster is generating data but not sending it to InfluxDB. When I look at the container logs for the eventer container within the Heapster pod, the logs look something like this:
E1020 18:43:20.006608 52 influxdb.go:147] Failed to create infuxdb: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp: lookup monitoring-influxdb on 10.254.0.10:53: read udp 172.16.23.2:43533->10.254.0.10:53: i/o timeout
I1020 18:43:30.000165 52 manager.go:95] Exporting 88 events
I1020 18:44:00.000173 52 manager.go:95] Exporting 29 events
W1020 18:44:20.000451 52 manager.go:108] Failed to events data to sink: InfluxDB Sink
E1020 18:44:20.075039 52 influxdb.go:147] Failed to create infuxdb: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp: lookup monitoring-influxdb on 10.254.0.10:53: read udp 172.16.23.2:60660->10.254.0.10:53: i/o timeout
I1020 18:44:30.000211 52 manager.go:95] Exporting 26 events
I1020 18:45:00.000169 52 manager.go:95] Exporting 23 events
W1020 18:45:20.000568 52 manager.go:108] Failed to events data to sink: InfluxDB Sink
E1020 18:45:20.004656 52 influxdb.go:147] Failed to create infuxdb: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp: lookup monitoring-influxdb on 10.254.0.10:53: read udp 172.16.23.2:53401->10.254.0.10:53: i/o timeout
I1020 18:45:30.000568 52 manager.go:95] Exporting 30 events
I1020 18:46:00.000188 52 manager.go:95] Exporting 26 events
W1020 18:46:20.000504 52 manager.go:108] Failed to events data to sink: InfluxDB Sink
E1020 18:46:20.074697 52 influxdb.go:147] Failed to create infuxdb: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp: lookup monitoring-influxdb on 10.254.0.10:53: read udp 172.16.23.2:49101->10.254.0.10:53: i/o timeout
I1020 18:46:30.000267 52 manager.go:95] Exporting 39 events
I1020 18:46:35.052332 52 influxdb.go:199] Created database "k8s" on influxDB server at "monitoring-influxdb:8086"
I1020 18:47:00.000173 52 manager.go:95] Exporting 64 events
I1020 18:47:30.000212 52 manager.go:95] Exporting 52 events
I1020 18:48:00.000234 52 manager.go:95] Exporting 43 events
I1020 18:48:30.070486 52 manager.go:95] Exporting 52 events
I1020 18:49:00.000340 52 manager.go:95] Exporting 40 events
I1020 18:49:30.000170 52 manager.go:95] Exporting 40 events
I1020 18:50:00.000205 52 manager.go:95] Exporting 57 events
I1020 18:50:30.000211 52 manager.go:95] Exporting 94 eventsAfter that there's just a series of events being exported. My first question is did Heapster successfully connect to the InfluxDB service or is it just sending data to http://monitoring-influxdb:8086 assuming that something will be collected? If it did connect, why is the k8s database in my InfluxDB empty?
I deployed my cluster using the ansible script in kubernetes/contrib/ansible and the yaml files were downloaded from somwhere (I don't know where). I pushed them up to a git repository here if you want to take a look.
The easiest way (IMO) to determine if it's collecting data is to look at the logs for the influxdb. In my environment (deployed from https://github.com/kubernetes/heapster/tree/master/deploy/kube-config/influxdb), the pods are all in the kube-system namespace:
kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
heapster-603813915-tggbr 1/1 Running 0 34d
kubernetes-dashboard-3313488171-mpn5b 1/1 Running 2 43d
monitoring-grafana-2175968514-zxgn1 1/1 Running 0 34d
monitoring-influxdb-1957622127-92r2w 1/1 Running 0 34dand the logs:
kubectl logs monitoring-influxdb-1957622127-92r2w -n kube-system
[httpd] 10.244.1.18 - root [23/Oct/2017:21:13:05 +0000] "POST /write?consistency=&db=k8s&precision=&rp=default HTTP/1.1" 204 0 "-" "heapster/v1.4.0" f5e74e91-b836-11e7-84d0-000000000000 121606 [httpd] 10.244.1.18 - root [23/Oct/2017:21:14:05 +0000] "POST /write?consistency=&db=k8s&precision=&rp=default HTTP/1.1" 204 0 "-" "heapster/v1.4.0" 19a8a32c-b837-11e7-84d1-000000000000 137213 [httpd] 10.244.1.18 - root [23/Oct/2017:21:15:05 +0000] "POST /write?consistency=&db=k8s&precision=&rp=default HTTP/1.1" 204 0 "-" "heapster/v1.4.0" 3d6eb415-b837-11e7-84d2-000000000000 125065 [httpd] 10.244.1.18 - root [23/Oct/2017:21:16:05 +0000] "POST /write?consistency=&db=k8s&precision=&rp=default HTTP/1.1" 204 0 "-" "heapster/v1.4.0" 6133e847-b837-11e7-84d3-000000000000 128300
The 204 responses are pretty clear that it's accepting data in this case. If your heapster isn't communicating with influxdb for some reason, then the answer is likely to be within the logs for the influxdb pod as well.