I'm building a dashboard in Grafana, using data from Prometheus, to monitor a namespace in a Kubernetes cluster. I need all this to see what happens during a load test.
Now I've spent half my day looking for information about the different metrics in Prometheus. I'v read through Prometheus docs and kube state metrics docs (which gets the data from our cluster) but I did not find any descriptions about which metric does what. I only can guess based on query results and examples found here and there, but that's slow and way more insecure than I'd like.
However, I've come upon this SO answer so I assume the quote must have been copied from somewhere. Anybody please?
I think you want to find "which metric does what" which is irrespective of kubernetes.
Prometheus will get data from exporter and store in its database, So you have to ask this question to your exporter which is exporting the matrix.
Eq. You can get basic CPU memory from node exporter. You can find all matrix exposed by Node exported in the below link.
https://github.com/prometheus/node_exporter
Now the question remains which is exporting which matrix that can be found from the target in Prometheus UI.
I found the answer for my question. yogesh's answer gave me the hint to have a look at exporters and I found the other half of the answer here.
So on Prometheus UI, there is a list of exporters and their scraped endpoints (Status > Targets). If I call one of the endpoints, the response contains a description and the type of each metric offered by the endpoint.
Call:
curl http://exporter.endpoint:9100/metricsSample from a response:
# HELP http_response_size_bytes The HTTP response sizes in bytes.
# TYPE http_response_size_bytes summary
http_response_size_bytes{handler="prometheus",quantile="0.5"} 16310
http_response_size_bytes{handler="prometheus",quantile="0.9"} 16326
http_response_size_bytes{handler="prometheus",quantile="0.99"} 16337
http_response_size_bytes_sum{handler="prometheus"} 1.46828673e+09
http_response_size_bytes_count{handler="prometheus"} 90835
# HELP node_arp_entries ARP entries by device
# TYPE node_arp_entries gauge
node_arp_entries{device="cali6b3dc03715a"} 1
node_arp_entries{device="eth0"} 15It wasn't trivial for me how to get this done. I logged into the cluster and I found curl didn't get any response for any of the endpoints. The solution was to add the base of the endpoint urls to the no_proxy variable (my machine and the server are both sitting behind the corporate proxy).
But anyway, this is how one can read the description of the Prometheus metrics.