I don't know if this is a bug/issue or question.

Proposal

Use case. Why is this important?

For monitoring multiple containers with resource/requests limits in pods on kubernetes.

Bug Report

What did you do?

I'm write a query to get a percentage of usage based on the maximum CPU usage and that we have max on the limits (resource and request) of the pod.

We have this problem affecting our query:

1. When we take a pod and it have 2 containers with configured resource/requets limits, it is not possible to take the value of resource/requests limits. 2. Show the value of the pod (resource/requests), but it can have multiple replicas.

max_over_time(sum(rate(container_cpu_usage_seconds_total{namespace="alpha",container_name!="POD", container_name!=""}[1m])) [1h:1s]) / on(pod) kube_pod_container_resource_requests_cpu_cores{namespace="alpha"}

Error executing query:found duplicate series for the match group {pod="heapster-65ddcb7b4c-vtl8j"} on the right hand-side of the operation: [{__name__="kube_pod_container_resource_requests_cpu_cores", container="heapster-nanny", instance="kubestate-alpha.internal:80", job="k8s-prod-http", namespace="alpha", node="ip-99-990-0-99.sa-east-1.compute.internal", pod="heapster-65ddcb7b4c-vtl8j"}, {__name__="kube_pod_container_resource_requests_cpu_cores", container="heapster", instance="kubestate-alpha.internal:80", job="k8s-alpha-http", namespace="alpha", node="ip-99-990-0-99.sa-east-1.compute.internal", pod="heapster-65ddcb7b4c-vtl8j"}];many-to-many matching not allowed: matching labels must be unique on one side.

We try solutions like: [Using group_left to calculate label proportions]

sum without (container) (rate(kube_pod_container_resource_requests_cpu_cores{pod="heapster-65ddcb7b4c-vtl8j"}[1m]))

But if the value is set in the container, the result of the query is 0. For not being able to calculate.

kube_pod_container_resource_requests_cpu_cores{pod="heapster-65ddcb7b4c-vtl8j"}

kube_pod_container_resource_requests_cpu_cores{container="heapster", instance="kubestate-alpha.internal:80", job="k8s-alpha-http", namespace="alpha", node="ip-99-990-0-99.sa-east-1.compute.internal", pod="heapster-65ddcb7b4c-vtl8j"} 0.084 kube_pod_container_resource_requests_cpu_cores{container="heapster-nanny", instance="kubestate-alpha.internal:80", job="k8s-alpha-http", namespace="alpha", node="ip-99-990-0-99.sa-east-1.compute.internal", pod="heapster-65ddcb7b4c-vtl8j"} 0.05

Standard output for the kube_pod_container_resource_requests_cpu_cores command

What did you expect to see?

The sum of what is set in the containers in the pod.

What did you see instead? Under which circumstances?

Prometheus UI

Environment

• System information:

  Linux 4.4.0-1096-aws x86_64

• Prometheus version:

  v2.15.2