K
Q

Prometheus - Kubernetes cluster metrics

3/30/2020

I have multiple kubernetes clusters metric data being scraped in prometheus. When i get metrics how can i differentiate the metrics from different clusters? I am not seeing any label that contains data regarding a specific cluster, so that i can filter out data of a particular cluster like below,

container_cpu_usage_seconds_total{cluster-name="abcde"}

Is there any way where i can add a label "cluster-name" in my kubernetes_sd_configs. I have seen, labels can be added in static_config but can't find anything related to kubernetes_sd_configs.

I tried using relabel_config like below,

 - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: cluster-name
        replacement: my-cluster

This did not get reflected in metrics. When i do it with already existing label like,

 - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: domainname
        replacement: my-cluster

Then the domain name value is getting changed. Am i missing any configuration here?

-- Sathish Kumar
kubernetes
prometheus
promql

Similar Questions

Firewall Rules between GKE and GAE
what is ther difference between {{- range .Values.xxx }} and {{ range .Values.xxx }} in helm
Connect to dynamically create new cluster on GKE
no space left on device, which is unexpected. MountVolume.SetUp failed for volume in kubernetes cluster

1 Answer

3/30/2020

You have to add that label yourself in your relabel_configs. Generally you do this on the intake cluster (i.e. the normal "cluster-level" ones) via a global relabel to add it, but it's also possible to add during the remote write if you're doing federation that way or via the jobs on the central cluster if you're doing a pull-based.

-- coderanger
Source: StackOverflow