For example, kubelet(cAdvisor) container_cpu_usage_seconds_total has value with some parameter (e.g. pod, namespace).
I wonder how to summarize this kind of values into Service(for example, CPU usage per service)? I understand that Service is a set of pods so that just aggregating these values per pod to service, but I do not know how?
Is there any aggregation method to Service? Or, process_cpu_seconds_total is a kind of aggregated value per service of 'container_cpu_usage_seconds_total'?
Thank you for your help!
In general, cAdvisor collects metrics about containers and doesn't know anything about Services. If you want to aggregate by Service, you need to manually select the metrics of the Pod that belong to this Service.
For example, if your cAdvisor metrics are in Prometheus, you can use this PromQL query:
sum(rate(container_cpu_usage_seconds_total{pod_name=~"myprefix-*"}[2m]))This adds up the CPU usages of all containers of all Pods that have a name starting with myprefix-.
Or if you have the Resource Metrics API enabled (i.e. the Metrics Server installed), you can query the CPU usage of a specific Pod (in fractions of a CPU core) with:
kubectl get --raw="/apis/metrics.k8s.io/v1beta1/namespaces/{namespace}/pods/{pod}"To get the total usage of a Service, you would need to iterate through all the Pods of the Service, extract the values, and add them together.
In general, Service is a Kubernetes concept and does not exist in cAdvisor, which is an independent project and just happens to be used in Kubernetes.
What about
sum(rate(container_cpu_usage_seconds_total{job="kubelet", cluster="", namespace="default", pod_name=~"your_pod_name.*"}[3m]))Taken from kubernetes-mixin