Intermittent timeout on Kubernetes services

1/12/2020

I was wondering if anyone can point me in the right direction as I've been pulling my hair out with this one.

So, I've got a couple of simple services running (through a NodePort) and for the most of time, they run fine. But every once in a while, we get a timeout on the services which I cannot explain.

I've been testing with the following commandline which will reproduce the issue quite fast.

while true; do clear && wget k8s-mn01:30088/CallBack/Status -O /dev/null && sleep 0.05; done

Once the issue presents itself, I will get the following response:

Resolving k8s-mn01 (k8s-mn01)... 192.168.123.101
Connecting to k8s-mn01 (k8s-mn01)|192.168.123.101|:30088... connected.
HTTP request sent, awaiting response...

I'm currently running a 3-node BareMetal cluster consisting of 1 master and 2 workers. Everything is installed on CentOS 7.

Kubernetes version 1.17 is installed with the flannel network.

At first (based on the information on the internet) I thought that this might be related to dropped packets which I would see in conntrack. But unfortunately, this is not the case.

One final thing which might be of interrest: When I try to access the service on the master node, it will always fail.

Can anyone give me pointers in the right direction?

Thanks in advance

-- Mekroebo
kubernetes
kubernetes-service
networking

0 Answers