I'm very new to Kubernetes, I just deployed few services on 2 node cluster.
Following are metrics
Nodes usage
kubectl -n services-namespace top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
gke-kubernetes-cluster-n-default-pool-aaaaaaaa-lvzm 151m 16% 1927Mi 72%
gke-kubernetes-cluster-n-default-pool-bbbbbbbb-6g58 266m 28% 2936Mi 110% Pods Usage
kubectl -n services-namespace top pods
NAME CPU(cores) MEMORY(bytes)
devv-aaa-service-2543400000-7ffbl 5m 269Mi
devv-bbb-service-2750127145-jfvw4 26m 276Mi
devv-aaa-service-2543400800-vlgmr 20m 285Mi
devv-aaa-service-2543000000-g82p1 4m 280Mi
devv-ccc-service-3204123401-7vv67 19m 283Mi
devv-aaa-service-2541023480-sd9cm 3m 284Mi
devv-ccc-service-3207045691-kjlfn 21m 268Mi
devv-aaa-service-2541204800-64q67 30m 270Mi What does it mean by MEMORY(%), is it current usage of Node. If so why nodes are taking lot of memory even though pods are not taking full Node memory. Cumulative PODS memory usage is around 2500Mi.
How to handle these Memory and CPU ?
What are the best practices to consider while using clusters.
Besides running pods , kubernetes nodes also runs OS systems daemons(like udev, sshd) that is required for kubernetes to function . For this the compute resources is reserved which is called as "system-reserved". Also, some more compute resources are reserved for kubernetes system daemons like container runtime and kubelet which is called as "kube-reserved".
So, Node Capacity=kube reserved+system reserved+eviction threshold+Allocatable(available for pods)
You can use "kubectl describe node and in the output look for "System Info" section which can give you detailed info on memory request and limits Also, please go over this link from kubernetes docs: https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/