I have the following Prometheus query and want the right side of my substraction to only subtract the container_memory_usage_bytes for each node on the left side of the substraction.
Query:
sum(kube_node_status_capacity_memory_bytes) by (node) - on (instance) group_left container_memory_usage_bytesThe left side of the subtraction works.
But for the whole query I get
Error executing query: found duplicate series for the match group;many-to-many matching not allowed: matching labels must be unique on one side
How can I make my query to match exactly the labels node == instance from left to right side?
The RHS has no instance label, so it's trying to match all those series to one on the LHS. You have to also add sum before listing container_memory_usage_bytes metric, - sum(container_memory_usage_bytes).
Please take a look here: prometheus-metrics.
Useful article: label-proportions.