K
Q

Prometheus only scrapes one pod

9/16/2019

I'm using Prometheus to scrape metrics from my pods. The application I'm interested in is replicated a couple of times with one service providing access. Prometheus uses this service to scrape the metrics. In my app the metrics are setup as follows:

import * as Prometheus from 'prom-client';

const httpRequestDurationMicroseconds = new Prometheus.Histogram({
    name: 'transaction_amounts',
    help: 'Amount',
    labelNames: ['amount'],
    buckets: [0, 5, 15, 50, 100, 200, 300, 400, 500, 10000],
});

const totalPayments = new Prometheus.Counter('transaction_totals', 'Total payments');

I'm using helm to install Prometheus and the scrape config looks like this:

prometheus.yml:
  rule_files:
    - /etc/config/rules
    - /etc/config/alerts

  scrape_configs:
    - job_name: prometheus
      static_configs:
        - targets:
          - localhost:9090
    - job_name: transactions
      scrape_interval: 1s
      static_configs:
        - targets:
          - transaction-metrics-service:3001

I can see the metrics inside prometheus, but it seems to be from just one pod. For example, in Prometheus, when I query for transaction_totals it gives:

enter image description here

I don't think that the instance label can uniquely identify my pods. What should I do to be able to query all pods?

-- Jeanluca Scaljeri
kubernetes
metrics
prometheus

Similar Questions

kubernetes: Failed to update endpoints warning
Return response from nginx-auth-url to the client request
How to set up an endpoint for Health check on Next.js?
See wrong client URL when listing the etcd member
Create JVM heapdump when K8s healthcheck restarts the pod - no OOM occur

2 Answers

9/16/2019

Instead of using a static_config that scrapes just one host, try using kubernetes_sd_configs Kubernetes Service Discovery as provided by Prometheus. Your config file would look something like this:

- job_name: 'kubernetes-pods'

  kubernetes_sd_configs:
  - role: pod

  relabel_configs:
  # only scrape when annotation prometheus.io/scrape: 'true' is set
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: (.+):(?:\d+);(\d+)
    replacement: ${1}:${2}
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name

and then add the annotation to your Kubernetes Deployment yaml config like this:

kind: Deployment

...

spec:
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "<< PORT OF YOUR CONTAINER >>"

You can see a full working example here.

-- Oliver
Source: StackOverflow
9/16/2019

add prometheus annotations to your service, since prom will only scrape a service that:

  • Exposes the exporter port
  • Has a prometheus.io/scrape: "true" annotation
  • Has a prometheus.io/port: "<exporter_port_here>" annotation

here is an official example

the scraped pod is probably prometheus itself

-- Efrat Levitan
Source: StackOverflow