I have a pod, that consists of just one container. This container has the following resource section:

  resources:
    limits:
      cpu: "4"
      memory: 58Gi
      nvidia.com/gpu: 1
    requests:
      cpu: 3500m
      memory: 55Gi

When I run this pod, it starts successfully and all is good until I try to copy a file there. If I run kubectl cp foo.txt my_pod:/app, I get the following error:

tar: This does not look like a tar archive
tar: Skipping to next header
tar: Exiting with failure status due to previous errors

However, if I omit any resource requirements(i.e. comment the resource section out), cp works as expected.

I did some investigation and found out that the problem is only present when nvidia.com/gpu: 1 is specified in the limits. Colleagues told me it is a known problem, but no one knows why is that.

Does anyone know? What is the best workaround here?