I'm trying to copy a local Python file to a running container on Kubernetes and it fails:

$ kubectl cp /path/to/file.py namespace/pod:/path/in/container/file.py
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
command terminated with exit code 2

I know the tar binary must be available in the container, and it is.

Does anyone know what's going on here and how I can solve this?

UPDATE:

After some more testing I can confirm that this only happens on nodes that run nvidia-docker rather than the normal docker. When piping things into kubectl exec on these nodes the stream is always empty.

So the following command yields an empty file in the pod running on a GPU-enabled node while the file is non-empty on other nodes without GPU support:

cat nonempty_file.txt | kubectl exec -i pod -- tee /home/jovyan/empty_file.txt 

This has been tested using the exact same image/container on both nodes.