Kubelet API /stats/summary/ usageNanoCores not correct

9/6/2016

Kubernetes version: 1.3.0 Docker version: 1.10.3 OS: CoreOS stable (1068.9.0) VM: m3.xlarge

On k8s node I am querying the kubelet /stats/summary to get CPU usage for node and am seeing unexpected results. Below is output of top command, I have run command to consume 100% of CPU.

Output from top Command

Here are the usageNanoCores and time values from calling /stats/summary/.

usageNanoCores: 3203229858
time: 2016-09-06T20:37:44Z
usageNanoCores: 3497892888
time: 2016-09-06T20:38:02Z
usageNanoCores: 3497892888
time: 2016-09-06T20:38:02Z
usageNanoCores: 3318318883
time: 2016-09-06T20:38:18Z
usageNanoCores: 3419241048
time: 2016-09-06T20:38:34Z
usageNanoCores: 3218012832
time: 2016-09-06T20:38:46Z
usageNanoCores: 3218012832
time: 2016-09-06T20:38:46Z
usageNanoCores: 3673825154
time: 2016-09-06T20:38:58Z
usageNanoCores: 3213540614
time: 2016-09-06T20:39:16Z
usageNanoCores: 3213540614
time: 2016-09-06T20:39:16Z
usageNanoCores: 3560073628
time: 2016-09-06T20:39:29Z
usageNanoCores: 3196742360
time: 2016-09-06T20:39:43Z
usageNanoCores: 3196742360
time: 2016-09-06T20:39:43Z
usageNanoCores: 3543065178
time: 2016-09-06T20:39:59Z
usageNanoCores: 3323870743
time: 2016-09-06T20:40:18Z
usageNanoCores: 3323870743
time: 2016-09-06T20:40:18Z
usageNanoCores: 3486174845
time: 2016-09-06T20:40:36Z
usageNanoCores: 3486174845
time: 2016-09-06T20:40:36Z
usageNanoCores: 3961347785
time: 2016-09-06T20:40:54Z
usageNanoCores: 3683434468
time: 2016-09-06T20:41:05Z
usageNanoCores: 3892752522
time: 2016-09-06T20:41:19Z
usageNanoCores: 3892752522
time: 2016-09-06T20:41:19Z
usageNanoCores: 3867897133
time: 2016-09-06T20:41:34Z
usageNanoCores: 3867897133
time: 2016-09-06T20:41:34Z
usageNanoCores: 3546368625
time: 2016-09-06T20:41:54Z

Since usageNanoCores is the "Total CPU usage (sum of all cores) averaged over the sample window" and I have 4 cores, shouldn't usageNanoCores be constistantly closer to 4000000000?

-- Brian
kubernetes

1 Answer

9/7/2016

top %cpu is percentage of a single core, so it looks like your busy process is single threaded and only using 100% of a single core. If you run 4 of them, I would expect usageNanoCores to be closer to the expected 4B.

-- Tim Allclair
Source: StackOverflow