Given a kubernetes cluster with: 1. prometheus 2. node-exporter 3. kube-state-metrics
I like to use the metric container_memory_usage_bytes
but select by deployment_name
instead of pod
.
Selectors like container_memory_usage_bytes{pod_name=~"foo-.+"}
if the deployment_name=foo
are great as long there is not a deployment with deployment_name=foo-bar
.
The same I'd like to achieve with the metric kube_pod_container_resource_limits_memory_bytes
.
Is there a way to achieve this?
The following PromQL query should return per-deployment memory usage in Kubernetes:
sum(
label_replace(
container_memory_usage_bytes{pod!=""},
"deployment", "$1", "pod", "(.+)-[^-]+-[^-]+"
)
) by (namespace,deployment)
The query works in the following way:
The container_memory_usage_bytes{pod!=""}
time series selector selects all the time series with the name container_memory_usage_bytes
and with non-empty pod
label. We need to filter out time series without pod
label, since such time series account for cgroups hierarchy, which isn't needed in this query. See this answer for more details on this.
The inner label_replace() extracts deployment name from pod
label and puts it to deployment
label. It expects that pod names are constructed with the following pattern: <deployment>-<some_suffix_1>-<some_suffix_2>
.
The outer sum() sums pod memory usage per each group with identical namespace
and deployment
labels.
TL;DR
There is no straightforward way to query prometheus by a deployment-name
.
You can query memory usage of a specific deployment by using deployment's labels.
Used query:
sum(
kube_pod_labels{label_app=~"ubuntu.*"} * on (pod) group_right(label_app) container_memory_usage_bytes{namespace="memory-testing", container=""}
)
by (label_app)
There is an awesome article which explains the concepts behind this query. I encourage you to read it:
I've included an explanation with example below.
The selector mentioned in the question:
container_memory_usage_bytes{pod_name=~"foo-.+"}
.+
- match any string but not an empty string
with pods like:
foo-12345678-abcde
- will match (deployment foo
)foo-deployment-98765432-zyxzy
- will match (deployment foo-deployment
) As shown above it will match for both pods and for both deployments.
For more reference:
As mentioned earlier, you can use labels from your deployment to pinpoint the resource used by your specific deployment.
Assuming that:
memory-testing
namespace:
ubuntu
with 3 replicas
ubuntu-additional
with 3 replicasapp: ubuntu
app: ubuntu-additional
1.18.X
Why do I specify Kubernetes version?
Kubernetes 1.16 will remove the duplicate
pod_name
andcontainer_name
metric labels from cAdvisor metrics. For the 1.14 and 1.15 release allpod
,pod_name
,container
andcontainer_name
were available as a grace period.
This means that you will need to substitute the parameters like:
pod
with pod_name
container
with container_name
To deploy Prometheus and other tools to monitor the cluster I used: Github.com: Coreos: Kube-prometheus
The pods in ubuntu
deployment are configured to generate artificial load (stress-ng
). This is done to show how to avoid situation where the used resources are doubled.
The resources used by pods in memory-testing
namespace:
$ kubectl top pod --namespace=memory-testing
NAME CPU(cores) MEMORY(bytes)
ubuntu-5b5d6c56f6-cfr9g 816m 280Mi
ubuntu-5b5d6c56f6-g6vh9 834m 278Mi
ubuntu-5b5d6c56f6-sxldj 918m 288Mi
ubuntu-additional-84bdf9b7fb-b9pxm 0m 0Mi
ubuntu-additional-84bdf9b7fb-dzt72 0m 0Mi
ubuntu-additional-84bdf9b7fb-k5z6w 0m 0Mi
If you were to query Prometheus with below query:
container_memory_usage_bytes{namespace="memory-testing", pod=~"ubuntu.*"}
You would get output similar to one below (it's cut to show only one pod for example purposes, by default it would show all pods with ubuntu
in name and in memory-testing
namespace):
container_memory_usage_bytes{endpoint="https-metrics",id="/kubepods/besteffort/podb96dea39-b388-471e-a789-8c74b1670c74",instance="192.168.0.117:10250",job="kubelet",metrics_path="/metrics/cadvisor",namespace="memory-testing",node="node1",pod="ubuntu-5b5d6c56f6-cfr9g",service="kubelet"} 308559872
container_memory_usage_bytes{container="POD",endpoint="https-metrics",id="/kubepods/besteffort/podb96dea39-b388-471e-a789-8c74b1670c74/312980f90e6104d021c12c376e83fe2bfc524faa4d4cee6553182d0fa2e007a1",image="k8s.gcr.io/pause:3.2",instance="192.168.0.117:10250",job="kubelet",metrics_path="/metrics/cadvisor",name="k8s_POD_ubuntu-5b5d6c56f6-cfr9g_memory-testing_b96dea39-b388-471e-a789-8c74b1670c74_0",namespace="memory-testing",node="node1",pod="ubuntu-5b5d6c56f6-cfr9g",service="kubelet"} 782336
container_memory_usage_bytes{container="ubuntu",endpoint="https-metrics",id="/kubepods/besteffort/podb96dea39-b388-471e-a789-8c74b1670c74/1b93889a3e7415ad3fa040daf89f3f6bc77e569d85069de518267666ede8e21c",image="ubuntu@sha256:55cd38b70425947db71112eb5dddfa3aa3e3ce307754a3df2269069d2278ce47",instance="192.168.0.117:10250",job="kubelet",metrics_path="/metrics/cadvisor",name="k8s_ubuntu_ubuntu-5b5d6c56f6-cfr9g_memory-testing_b96dea39-b388-471e-a789-8c74b1670c74_0",namespace="memory-testing",node="node1",pod="ubuntu-5b5d6c56f6-cfr9g",service="kubelet"} 307777536
In this point you will need to choose which metric you will be using. In this example I used the first one. For more of a deep dive please take a look on this articles:
If we were to aggregate this metrics with sum (QUERY) by (pod)
we would have in fact doubled our reported used resources.
Dissecting the main query:
container_memory_usage_bytes{namespace="memory-testing", container=""}
Above query will output records with used memory metric for each pod. The container=""
parameter is used to get only one record (mentioned before) which does not have container
parameter.
kube_pod_labels{label_app=~"ubuntu.*"}
Above query will output record with pods and it's labels with regexp of ubuntu.*
kube_pod_labels{label_app=~"ubuntu.*"} * on (pod) group_right(label_app) container_memory_usage_bytes{namespace="memory-testing", container=""}
Above query will match the pod
from kube_pod_labels
with pod
of container_memory_usage_bytes
and add the label_app
to each of the records.
sum (kube_pod_labels{label_app=~"ubuntu.*"} * on (pod) group_right(label_app) container_memory_usage_bytes{namespace="memory-testing", container=""}) by (label_app)
Above query will sum the records by label_app
.
After that you should be able to get the query that will sum the used memory by a label (and in fact a Deployment).
As for:
The same I'd like to achieve with the metric
kube_pod_container_resource_limits_memory_bytes
.
You can use below query to get the memory limit for the deployment tagged with labels as in previous example, assuming that each pod in a deployment is having the same limits:
kube_pod_labels{label_app="ubuntu-with-limits"} * on (pod) group_right(label_app) kube_pod_container_resource_limits_memory_bytes{namespace="memory-testing", pod=~".*"}
You can apply functions like avg()
,mean()
,max()
on this query to get the single number that will be your memory limit:
avg(kube_pod_labels{label_app="ubuntu-with-limits"} * on (pod) group_right(label_app) kube_pod_container_resource_limits_memory_bytes{namespace="memory-testing", pod=~".*"}) by (label_app)
Your memory limits can vary if you use VPA
. In that situation you could show all of them simultaneously or use the avg()
to get the average for all of the "deployment".
As a workaround to above solutions you could try to work with regexp like below:
container_memory_usage_bytes{pod=~"^ubuntu-.{6,10}-.{5}"}