I am running prometheus in my kubernetes cluster. I have the following system in kubernetes:
I have 4 nodes. I want to calculate free memory. I want to have the summation of those four nodes. Then I want to find the maximum over 1 day. So, for example,
at time=t1 node1: 500 MB node2: 600 MB node3: 200 MB node4: 300 MB Total = 1700 MB
at time=t2 node1: 400 MB node2: 700 MB node3: 100 MB node4: 200 MB Total = 1300 MB
at time=t3 node1: 600 MB node2: 800 MB node3: 1200 MB node4: 1300 MB Total = 3900 MB
at time=t4 node1: 100 MB node2: 200 MB node3: 300 MB node4: 400 MB Total = 1000 MB
So, The answer to my query should be 3900 MB. I am not able to do max_over_time for the sum.
I have done like this(which is not working at all):
max_over_time(sum(node_memory_MemFree)[2m])Since version 2.7 (Jan 2019), Prometheus supports sub-queries:
max_over_time( sum(node_memory_MemFree_bytes{instance=~"foobar.*"})[1d:1h] )(metric for the past 2 days, with a resolution of 1 hour.)
Read the documentation for more informations on using recording rules: https://prometheus.io/docs/prometheus/latest/querying/examples/#subquery
However, do note the blog recommendation:
Epilogue
Though subqueries are very convenient to use in place of recording rules, using them unnecessarily has performance implications. Heavy subqueries should eventually be converted to recording rules for efficiency.
It is also not recommended to have subqueries inside a recording rule. Rather create more recording rules if you do need to use subqueries in a recording rule.
The use of recording rules is explained in brian brazil article: https://www.robustperception.io/composing-range-vector-functions-in-promql/
This isn't possible in one expression, you need to use a recording rule for the intermediate expression. See https://www.robustperception.io/composing-range-vector-functions-in-promql/