Issue with overriding labels in prometheus

11/25/2019

I installed stable/prometheus from helm. By default the job_name kubernetes-service-endpoints contains node-exporter and kube-state-metrics as component label. I added the below configuration in prometheus.yml to include namespace, pod and node labels.

      - source_labels: [__meta_kubernetes_namespace]
        separator: ;
        regex: (.*)
        target_label: namespace
        replacement: $1
        action: replace
      - source_labels: [__meta_kubernetes_pod_name]
        separator: ;
        regex: (.*)
        target_label: pod
        replacement: $1
        action: replace
      - source_labels: [__meta_kubernetes_pod_node_name]
        separator: ;
        regex: (.*)
        target_label: node
        replacement: $1
        action: replace

kube_pod_info{component="kube-state-metrics"} already had namespace, pod and node labels and hence exported_labels were generated. And the metric node_cpu_seconds_total{component="node-exporter"} now correctly has labels namespace, pod and node.

To have these labels correctly, I need to have these 3 labels present in both the above metric names. To achieve that I can override value of exported_labels. I tried adding the below config but to no avail.

      - source_labels: [__name__, exported_pod]
        regex: "kube_pod_info;(.+)"
        target_label: pod
      - source_labels: [__name__, exported_namespace]
        regex: "kube_pod_info;(.+)"
        target_label: namespace
      - source_labels: [__name__, exported_node]
        regex: "kube_pod_info;(.+)"
        target_label: node

Similar approach was mentioned here. I can't see the issue with my piece of code. Any directions to resolve would be very helpful.

Updated - (adding complete job)

    - job_name: kubernetes-service-endpoints
      kubernetes_sd_configs:
      - role: endpoints

      metric_relabel_configs:
      - source_labels: [__name__, exported_pod]
        regex: "kube_pod_info;(.+)"
        target_label: pod
      - source_labels: [__name__, exported_namespace]
        regex: "kube_pod_info;(.+)"
        target_label: namespace
      - source_labels: [__name__, exported_node]
        regex: "kube_pod_info;(.+)"
        target_label: node

      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scrape
      - action: replace
        regex: (https?)
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scheme
        target_label: __scheme__
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_path
        target_label: __metrics_path__
      - action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        source_labels:
        - __address__
        - __meta_kubernetes_service_annotation_prometheus_io_port
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        regex: (.*)
        replacement: $1
        separator: ;
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: node

And the result from promql

kube_pod_info results

-- Ankit Nayan
kubernetes
monitoring
prometheus

1 Answer

11/27/2019

So your goal is to rename the metric labels exported_pod to pod, etc, for the kube_pod_info metric?

In that case, you need metric relabelling which is done when metrics are fetched from targets:

- job_name: 'kubernetes-service-endpoints'

  kubernetes_sd_configs:
    - role: endpoints

  metric_relabel_configs:
    - source_labels: [__name__, exported_pod]
      regex: "kube_pod_info;(.+)"
      target_label: pod
    - source_labels: [__name__, exported_namespace]
      regex: "kube_pod_info;(.+)"
      target_label: namespace
    - source_labels: [__name__, exported_node]
      regex: "kube_pod_info;(.+)"
      target_label: node
  relabel_configs:
    # Insert the same what you have so far

Background:

Normal relabelling (relabel_configs) is applied at service discovery time to the target labels that are automatically discovered by the service discovery process. It defines the definitive target labels. At scrape time, target labels are added to the metric labels of all the metrics from the target. Normal relabelling can be only used to work on labels of a target after service discovery, which are generally meta labels starting with __.

Metric relabelling (metric_relabel_configs) is applied to the metric labels at scrape time. So, this can be used to rename labels that are defined by the applications exposing the metrics themselves.

-- weibeld
Source: StackOverflow