I monitor the eks cluster using both the kubernetes api and the kube state metrics remotely using prometheus. in Kubernetes api we have a metrics container_cpu_usage_seconds_total which gives the cpu usage of the pod. is there a similar metrics in kube-state-metrics which can give the cpu usage. Actually I'm trying to get the cluster cpu usage which is totally different from kubernetes api and kube-state-metrics following are the calculations.
kube-state-metrics:
sum(kube_pod_container_resource_requests_cpu_cores) / sum(kube_node_status_allocatable_cpu_cores) * 100 - This gives 101%
whereas the kube-state-metrics gives 12% which looks accurate to me.
kubernetes-api:
sum (rate (container_cpu_usage_seconds_total{id="/",kubernetes_io_hostname=~"^$Nodequot;, job=~"$jobquot;}[5m])) / sum (machine_cpu_cores{kubernetes_io_hostname=~"^$Nodequot;, job=~"$jobquot;}) * 100
I don't think there's any metric in kube-state-metric which gives cpu usage compared to kubernetes-api
Thanks in advance.
No there is no (one) specific metric for cpu usage per container in kube_state_metric.
Value you got: sum(kube_pod_container_resource_requests_cpu_cores) / sum(kube_node_status_allocatable_cpu_cores) * 100 = 101 may be wrong because metrics like kube_node_status_allocatable_cpu_cores and kube_pod_container_resource_requests_cpu_cores are marked as DEPRECIATED.
At the same time take notice that there is metric like kube_pod_container_resource_limits_cpu_cores. Your container could have set resource limit, that's why probably your result exceed 100 %. If you have limit set per container check if resource limit is lower that resource request and then your calculation should looks like: [sum(kube_pod_container_resource_requests_cpu_cores) - sum(kube_pod_container_resource_limits_cpu_cores)]/ sum(kube_node_status_allocatable_cpu_cores) * 100.
Take a look for every resource metrics in kube_state_metrics for container and nodes: node_metrics, pod_container_metrics.