K
Q

Prometheus many-to-many problem for kube cronjobs

1/4/2019

Hy there,

I'm trying to configure Kubernetes Cronjobs monitoring & alerts with Prometheus. I found this helpful guide

But I always get a many-to-many matching not allowed: matching labels must be unique on one side error.

For example, this is the PromQL query which triggers this error:

max( kube_job_status_start_time * ON(job_name) GROUP_RIGHT() kube_job_labels{label_cronjob!=""} ) BY (job_name, label_cronjob)

The queries by itself result in e.g. these metrics

kube_job_status_start_time: kube_job_status_start_time{app="kube-state-metrics",chart="kube-state-metrics-0.12.1",heritage="Tiller",instance="REDACTED",job="kubernetes-service-endpoints",job_name="test-1546295400",kubernetes_name="kube-state-metrics",kubernetes_namespace="monitoring",kubernetes_node="REDACTED",namespace="test-develop",release="kube-state-metrics"}

kube_job_labels{label_cronjob!=""}: kube_job_labels{app="kube-state-metrics",chart="kube-state-metrics-0.12.1",heritage="Tiller",instance="REDACTED",job="kubernetes-service-endpoints",job_name="test-1546295400",kubernetes_name="kube-state-metrics",kubernetes_namespace="monitoring",kubernetes_node="REDACTED",label_cronjob="test",label_environment="test-develop",namespace="test-develop",release="kube-state-metrics"}

Is there something I'm missing here? The same many-to-many error happens for every query I tried from the guide. Even constructing it by myself from ground up resulted in the same error. Hope you can help me out here :)

-- hajowieland
kube-state-metrics
kubernetes
prometheus
promql

Similar Questions

Kubernetes: The code change does not appear, is there a way to sync?
GCP k8s - from a cluster to another cluster with mongodb
AKS Upgrade "Operation failed with status: 200. Details: Resource state Failed"
MySQL pod crashes when connecting to it from other pods (minikube problem)

3 Answers

1/9/2019

I dug into this issue a bit more, and I guess the root cause of it is within this one-to-many vector matching expression:

kube_job_status_start_time * ON(job_name) GROUP_RIGHT() kube_job_labels{label_cronjob!=""}

where the group modifier "GROUP_RIGHT()" suggests, that each vector element from the left side (kube_job_status_start_time) can match with multiple elements on the right side (kube_job_labels), based on common label (job_name). The thing is that we are really dealing here with many-to-many matching, as each vector element from right side can match also multiple elements from left vector as well:

enter image description here enter image description here

I think that what we are missing here is the way to uniquely identify exported Job objects from K8S by Prometheus. The author of this blog post, mentions about this feature in his setup:

...Prometheus resolves this collision of label names by including the raw metric’s label as an exported_job label...

In my case I don't get this extra label from Prometheus when installed via helm (stable/prometheus-operator).

-- Nepomucen
Source: StackOverflow
2/26/2019

Replacing kube_job_status_start_time with max(kube_job_status_start_time) by (job_name) will aggregate out any duplicates and should resolve the error.

The resulting query will look like this

       max(
            max(kube_job_status_start_time) by (job_name)
            * ON(job_name) GROUP_RIGHT()
            kube_job_labels{label_cronjob!=""}
          ) BY (job_name, label_cronjob)
-- matcha
Source: StackOverflow
4/15/2020

In my case I don't get this extra label from Prometheus when installed via helm (stable/prometheus-operator).

You need to configure it in Prometheus. It calls: honor_labels: false

# If honor_labels is set to "false", label conflicts are resolved by renaming
# conflicting labels in the scraped data to "exported_<original-label>" (for
# example "exported_instance", "exported_job") and then attaching server-side
# labels.

So you have to configure your prometheus.yaml file - config with option honor_labels: false

# Setting honor_labels to "true" is useful for use cases such as federation and
# scraping the Pushgateway, where all labels specified in the target should be
# preserved

Anyway if I have it like this (I have now exported_jobs), still can't do proper query, but I guess is still because of my LHS.

Error executing query: found duplicate series for the match group 
{exported_job="kube-state-metrics"} on the left hand-side of the operation: 
[{__name__=
-- maitza
Source: StackOverflow