Kubernetes Pods not using CPU more than 1m

12/31/2019

My cluster is in AKS with 5 Nodes of size Standard_D4s_v3 and with K8s version 1.14.8.

As soon as a pod is started/restarted it shows Running (kubectl get pods) and up until the pods are in Running state the CPU usage shows 150m or as much as they require.

But when I top it (kubectl top po) after a pod has moved to Running state, the specific pod shows only 1m CPU usage, but Memory usage is where they should be and the service is down as well.

Kubectl logs -f (pod_name) returns nothing but I can ssh into the pods(kubectl exec -it ....)

-- Shahnewaz Ul Islam Chowdhury
azure
kubernetes

1 Answer

12/31/2019

It's totally normal behavior, if You create pod it needs more CPU resources to create it, once it's created it doesn't need that much resources anymore.

You can always use cpu/memory limits and resources, more about it with examples how to do it here

Pod CPU/Memory requests define a set amount of CPU and memory that the pod needs on a regular basis. When the Kubernetes scheduler tries to place a pod on a node, the pod requests are used to determine which node has sufficient resources available for scheduling. Not setting a pod request will default it to the limit defined. It is very important to monitor the performance of your application to adjust these requests. If insufficient requests are made, your application may receive degraded performance due to over scheduling a node. If requests are overestimated, your application may have increased difficulty getting scheduled.


Pod CPU/Memory limits are the maximum amount of CPU and memory that a pod can use. These limits help define which pods should be killed in the event of node instability due to insufficient resources. Without proper limits set pods will be killed until resource pressure is lifted. Pod limits help define when a pod has lost of control of resource consumption. When a limit is exceeded, the pod is prioritized for killing to maintain node health and minimize impact to pods sharing the node. Not setting a pod limit defaults it to the highest available value on a given node. Don't set a pod limit higher than your nodes can support. Each AKS node reserves a set amount of CPU and memory for the core Kubernetes components. Your application may try to consume too many resources on the node for other pods to successfully run. Again, it is very important to monitor the performance of your application at different times during the day or week. Determine when the peak demand is, and align the pod limits to the resources required to meet the application's max needs.

-- jt97
Source: StackOverflow