Remove default vars from custom K8s prometheus exporter

8/20/2020

I'm starting playing with custom exporters(Using kubernetes,grafana and prometheus) and I have a problem. I managed to expose my metrics correctly but every time I kill the pod that is sending them, the vars change and grafana plots a different colour(like a new info).

Is there any way to only keep app as var, I think that the problem are the vars that change(pod name and ip)?

MyMetric{app="prometheus-export-mymetric",instance="172.26.32.69:3000",job="kubernetes-pods",kubernetes_namespace="default",kubernetes_pod_name="prometheus-export-mymetric-66694564b8-r4pqc",pod_template_hash="66694564b8"}

Thanks in advance.

-- gixnex
grafana
kubernetes
prometheus
prometheus-node-exporter

1 Answer

8/20/2020

instead of kubernetes_pod_name you shall use pod labels that stay the same after redeploying.

in prometheus config we are using something like this:

 - job_name: kubernetes-pods
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    separator: ;
    regex: "true"
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    separator: ;
    regex: (.+)
    target_label: __metrics_path__
    replacement: $1
    action: replace
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    separator: ;
    regex: ([^:]+)(?::\d+)?;(\d+)
    target_label: __address__
    replacement: $1:$2
    action: replace
  - separator: ;
    regex: __meta_kubernetes_pod_label_(.+)
    replacement: $1
    action: labelmap
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: kubernetes_namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: kubernetes_pod_name
    replacement: $1
    action: replace
-- Vlad Ulshin
Source: StackOverflow