kubernetes 1.11 kube-apiserver use all cpu

4/25/2019

kubernetes 1.11 kube-apiserver use all cpu, but all components are healthy

AME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok                   
scheduler            Healthy   ok                   
etcd-1               Healthy   {"health": "true"}   
etcd-0               Healthy   {"health": "true"}   
etcd-2               Healthy   {"health": "true"} 

top command show apiserver used 100% CPU

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                      
 1171 root      20   0 1151124 781368  77040 S 115.0   9.6 143:34.29 kube-apiserver                               
 7903 root      20   0 10.732g 770004  46784 S   5.0  9.4   7:39.33 etcd                                         
 8957 root      20   0 1365948 121832  73864 S   3.7  1.5   2:42.37 kubelet                                      
10369 root      20   0   44012  31584  20276 S   1.7  0.4   1:43.64 calico-felix 

here is my nodes

NAME            STATUS   ROLES                      AGE   VERSION
10.27.233.109   Ready    controlplane,etcd,worker   58d   v1.11.6
10.27.234.189   Ready    controlplane,etcd,worker   58d   v1.11.6
10.27.236.195   Ready    worker                     47d   v1.11.6
10.27.5.23      Ready    worker                     58d   v1.11.6
10.81.58.120    Ready    worker                     6d    v1.11.6
master1         Ready    controlplane,etcd          2h    v1.11.6

just wonder why it occurs

-- faker
kubernetes

1 Answer

6/17/2019

The 115% that you saw might not actually mean that you run out of recources. Top Command shows CPU usage as a percentage of a single CPU by default. That way it is possible for you to see more than 100% usage if you have more than 1 core. For example: system with 2 cores will have a maximum of 200% CPU usage. Press I (shift + i) while top is running to shift between Irix and Solaris mode.

If that is not the case here I might have some suggestions:

  1. Check the logs to see what exacly is happening:

    • /var/log/kube-apiserver.log
    • journalctl -xe -
    • /etc/kubernetes/manifests/kube-apiserver.yaml (might also be usefull)
  2. Restart the kube-apiserver if needed.

Please let me know if that helped.

-- OhHiMark
Source: StackOverflow