I am running a Kubernetes v1.16.0 cluster of one master and three nodes over virtual machines that use Ubuntu 18.04. Installation has been done using Kubespray and virtual images were more or less clean. Also, I have to say that I have been using Kubernetes for years and this is not the first installation I do, so I think that the process I have followed during the installation is correct and has supposed no problems by itself.
What I am facing is that, even when the nodes themselves have no problem communicating with each other, containers running in different nodes are unable to handle messaging correctly. This results in messages that stop after some bytes have been sent and do not try to reconnect never after it. I think this is better explained with examples.
If I serve something over HTTP in node1 (an Apache2 website, for example) and curl it from node2, I get all the data with no problem. Communication is handled perfectly. Also, inside of node1 I can replicate the same between pod1 and pod2. This means that I can use one pod as a server and another one as a client and there is no problem with messaging. However, the problem appears when I try to communicate pod1.node1 and pod1.node2.
To understand this let's assume I have two kind of pods: server-pod and client-pod. Once deployed, there is one server-pod in node1 and two client-pod, one in node1 and the other one in node2. This way, I send curl GET request to server-pod.node1/info and, if I am in node1 the answer I get is something like { "name":"John", "age":30, "car":null }. On the other side, if I sent the curl request from node2, what I get is something like { "name":"John", "age":30, "c and it just gets blocked there, like waiting for more data. Of course these JSON are longer and are not cut in the first line. I guess that size is related to the MTU.
Additionally, if I query server-pod.node1 from client-pod.node2 and the response is small enough, communications happens with no problem. The problems seems to appear after some specific response that, again, I think it is related to MTU. I have tried to deploy all the pods of the network using hostNetwork: true flag to mitigate this problem, but then I have to face another one, DNS resolution. If I use host network DNS resolution is done over 8.8.8.8 and client-pod.node2 is unable to find server-pod.node1.
Anyone has any idea about why is this happening? I know it is hard to explain but I hope the problem is explained clearly enough.