K
Q

How to scrape pod level info using prometheus kubernetes?

1/18/2017

I am trying to scrape pod level info using prometheus kubernetes. Here is the config i am using:

 - job_name: 'kubernetes-pods'

  kubernetes_sd_configs:
  - api_servers:
    - 'https://kubernetes.default'
    role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: (.+):(?:\d+);(\d+)
    replacement: ${1}:${2}
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_pod_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name

But i don't see any info on grafana. Do I need to make any changes in my apps? snapshot

-- nocturnal
kubernetes
kubernetes-health-check
prometheus

Similar Questions

Profiling Java application in kubernetes
Deploy nginx official image, the volumn mount incorrectly
How to force https on ingress?
Vitess guestbook example not working in minikube

3 Answers

1/18/2017

you should make sure that the data which your Grafana graphs use is actually present in Prometheus.

I assume you're using this default Grafana dashboard.

Your Grafana query might look like this:

sum (container_memory_working_set_bytes{pod_name=~"^$Pod
quot;}) / sum (machine_memory_bytes{kubernetes_io_hostname=~"^$Node
quot;}) * 100

So you'd have to make sure that the container_memory_working_set_bytes metric is recorded and that it has a field pod_name.

In case the metric is not recorded you'd have to do some digging in the logs of prometheus to check whether it is able to crawl the API.

In case the metric is there but the labels are wrong you could either adjust the query in Grafana or add another relabel_config in Prometheus. See official documentation.

-- pagid
Source: StackOverflow
1/18/2017

With that configuration the first action asks that the pod be annotated with prometheus.io/scrape=true. Have you set that annotation on the pods in question?

-- brian-brazil
Source: StackOverflow
7/4/2017

Be aware,

there is a configuration mistake here: __meta_kubernetes_pod_namespace does not exist and should be __meta_kubernetes_namespace

Best regards,

Bart

-- Boeboe
Source: StackOverflow