multiple dynamic discovery labels using Prometheus

5/20/2020

i am trying to discover both pods and nodes on the same job using kubernetes_sd_configs and use their labels together. i have blackbox-exporter on multiple pods in my cluster on different nodes, my goal is to monitor each other. but i am having trouble with the relabelling. im trying to achieve something like this:

http://<pod-ip1>:8082/metrics?instance=<node-ip1>
http://<pod-ip2>:8082/metrics?instance=<node-ip1>
http://<pod-ipN>:8082/metrics?instance=<node-ip1>
.
.
http://<pod-ip1>:8082/metrics?instance=<node-ip2>
http://<pod-ip2>:8082/metrics?instance=<node-ip2>
http://<pod-ipN>:8082/metrics?instance=<node-ip2>
.
.

my current configuration looks like this but im missing the pod URL:

- job_name: 'kubernetes-pods'
        metrics_path: /probe
        params:
          module: [ping]

        kubernetes_sd_configs:
        - role: node

        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: <pod_should_be_here>:9115
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __param_target
        - source_labels: [__param_target]
          target_label: instance
-- tamirh
kubernetes
prometheus
prometheus-blackbox-exporter

0 Answers