K
Q

Can't transfer files between kubernetes pods running on different machines

5/18/2020

For over a year I have my static website deployed in a Kubernetes cluster having only one node (basically respecting Scenario 1 below).

These days I added another node to my cluster and everything works correctly, except one weird detail which blocked me publishing my website.

To exemplify, let's say I'm in pod1's shell running on node node1.

Scenario 1. If my website is running in a pod on the same node1, then

// pod1

$ wget <static-website-pod-IP>:PORT/some-file.html

successfully fetches some-file.html.

Scenario 2. In contrast, if my website is running in a pod on node2, then the same command

// pod1

$ wget <static-website-pod-IP>:PORT/some-file.html

  • successfully fetches some-file.html if the file has < ~1.4KB (empirically determined).

  • blocks forever if some-file.html > ~ 1.4KB.

For the static website I tried multiple nginx versions and configs and also a python dummy server.

I also tested with Kubernetes latest v1.18 and latest v1.16.

Any idea why I just can't download larger files from pods running on a different node?

-- svecax
kubernetes

Similar Questions

How to access kafka installed outside kubernetes cluster from a service provisioned inside a kubernetes cluster
Check if label exists in Kubernetes config for elk using puppet
Mount volume in k8s cluster - Connection Refused
Disadvantages of local Spring Cloud Data Flow installation for spring batch jobs

3 Answers

5/18/2020

In my opinion a better way for copying files from pods would be using kubectl cp:

kubectl cp <file-spec-src> <file-spec-dest>

POD in a specific container

kubectl cp <file-spec-src> <file-spec-dest> -c <specific-container>

Copy /tmp/foo local file to /tmp/bar in a remote pod in namespace

kubectl cp /tmp/foo <some-namespace>/<some-pod>:/tmp/bar

Copy /tmp/foo from a remote pod to /tmp/bar locally

kubectl cp <some-namespace>/<some-pod>:/tmp/foo /tmp/bar
-- omricoco
Source: StackOverflow
5/18/2020

Since it is stuck at 1,4 kB, seems like your kubernetes networking plugin is misconfigured, as it transfers exactly one MSS/MTU.

-- Anton Matsiuk
Source: StackOverflow
5/19/2020

I managed to solve the issue starting from @anton-matsiuk 's hint.

The MTU used by the used network interface is 1450:

$ ip link show

...
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 86:00:00:53:4d:50 brd ff:ff:ff:ff:ff:ff
...

So the solution was to modify calico's ConfigMap's default param veth_mtu: "1440" to veth_mtu: "1430", according to this table.

Thank you, @anton-matsiuk.

-- svecax
Source: StackOverflow