Unable to relabel Prometheus pod scrape label __address__ if pod doesn't declare containerport

8/13/2021

According to Prometheus documentation:

"If a container has no specified ports, a port-free target per container is created for manually adding a port via relabeling."

However, it seems in my setup, pods with containerports have their address label converted as expected, but those without containerports are left "portless". Is this common/accepted behavior, or is there something I should change in order to apply __meta_kubernetes_pod_annotation_prometheus_io_port as the address port? Thanks!

    additionalScrapeConfigs:
    - job_name: 'perpod-metricsscrape'
      scrape_interval: 5s
      metrics_path: /metrics
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
      - source_labels: [ __meta_kubernetes_pod_annotation_prometheus_io_scrape ]
        action: keep
        regex: true
      - source_labels: [ __meta_kubernetes_pod_annotation_prometheus_io_path ]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [ __address__, __meta_kubernetes_pod_annotation_prometheus_io_port ]
        action: replace
        regex: (.+)(?::\d+);(\d+)
        replacement: ${1}:${2}
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)

BTW, I altered the regex slightly from most examples floating around, thinking they weren't matching an address without the port (is as is the case with the aforementioned containerport-less pods) - before relabeling: _ address _="1.2.3.4"

-- Rod LN
kubernetes
prometheus

1 Answer

8/16/2021

Found the issue in the regex, and replacement... I now match port-less _ address , capture optional port, and replace with _ meta_kubernetes_pod_annotation_prometheus_io_port:

   regex: ([^:]+)(:\d+)?;(\d+)
   replacement: ${1}:${3}
-- Rod LN
Source: StackOverflow