Prometheus scrape config for multiple metric endpoints per pod

2/6/2018

We have a Kubernetes Pod which provides multiple metric endpoints (:3093/metrics and :9113/metrics), but it also has an port which doesn't provide any metric (:80).

TL;DR: Is is possible to scrape only the ports 3093 and 9113?


We are using the example configuration for the scrape configuration:

- job_name: 'kubernetes-pods'
  tls_config:
    insecure_skip_verify: true
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name

These endpoints get properly scraped, when enabling scraping with this annotation:

  annotations:
    prometheus.io/scrape: "true"

But this has the issue, that it also scrapes port :80, which it shouldn't.

-- svenwltr
kubernetes
prometheus

1 Answer

2/13/2018

We created an exporter which merges the output of multiple other exporters. It is very alpha, but it works for us, now.

https://github.com/rebuy-de/exporter-merger

-- svenwltr
Source: StackOverflow