High disk IO in Kubernetes nodes

8/23/2018

Deployed 3 node K8s cluster on CentOS 7 VMs(vsphear) with kubeadm.

Master - 4GB    
Minion1 - 6GB RAM
Minion2 - 6GB RAM
K8s Version - v1.11.2

Before starting kubelet daemon on each node, I did swapoff -a to disable swap memory which is recommended for K8s

Deployed Spinnaker on K8s cluster and I see all spinnaker components are spring boot java applications.

I noticed that after some time, I'm losing ssh connection(able to ping) and nodes become NotReady state. After force restart of nodes, I started kubelet(after swapoff -a) while looking I/O states, iotop. Then I saw that java application doing lot of Disk Reads but not much CPU and memory. Also noticed that, sometimes nodes alive for long times(Hours). Verified same scenario few times in vsphear UI graph(sudden increase of disk reads).

Now my question is, too many disk reads/IO because of swapoff -a? or java application doesn't like disabling swap memory? I see some similar issue in k8s github repo.

enter image description here

Any help would be appreciated.

-- Veerendra Kakumanu
java
kubernetes
spinnaker

0 Answers