I have a pod which downloads a new machine learning model every 5 minutes, which causes the memory to grow indefinitely, and the pod is OOMkilled.

When running the application without docker locally, it uses about 2gb of memory, burst to about 3 during the download, and drops back to 2gb. When running in the cluster, the memory usage grows approximately 1gb per download, and never falls back down.

I specifically rm the downloaded models after the download. After seeing this question, I tried storing the downloads and loading the model from an empty-dir volume, without any effect.

Could anybody shine a light on what’s going on here?

Edit:

output from ps -Af | grep docker from running the image locally with

docker run -v tmp:/app/tmp -it [image_name]

0  2393     1   0 Mon09AM ??         0:00.01 /Library/PrivilegedHelperTools/com.docker.vmnetd
501  7020  6935   0 Mon01PM ??         0:04.47 /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper /Users/andreas.schafferer/.vscode/extensions/peterjausovec.vscode-docker-0.3.0/node_modules/vscode-languageclient/lib/utils/electronForkStart /Users/andreas.schafferer/.vscode/extensions/peterjausovec.vscode-docker-0.3.0/node_modules/dockerfile-language-server-nodejs/lib/server.js --node-ipc --node-ipc --clientProcessId=6935
501 53333  2377   0 10:09AM ??         0:00.12 /Applications/Docker.app/Contents/MacOS/com.docker.supervisor -watchdog fd:0
501 53334 53333   0 10:09AM ??         0:00.06 com.docker.osxfs serve --address fd:3 --connect vms/0/connect --control fd:4 --log-destination asl
501 53335 53333   0 10:09AM ??         0:03.38 com.docker.vpnkit --ethernet fd:3 --port fd:4 --diagnostics fd:5 --pcap fd:6 --vsock-path vms/0/connect --host-names host.docker.internal,docker.for.mac.host.internal,docker.for.mac.localhost --gateway-names gateway.docker.internal,docker.for.mac.gateway.internal,docker.for.mac.http.internal --vm-names docker-for-desktop --listen-backlog 32 --mtu 1500 --allowed-bind-addresses 0.0.0.0 --http /Users/andreas.schafferer/Library/Group Containers/group.com.docker/http_proxy.json --dhcp /Users/andreas.schafferer/Library/Group Containers/group.com.docker/dhcp.json --port-max-idle-time 300 --max-connections 2000 --gateway-ip 192.168.65.1 --host-ip 192.168.65.2 --lowest-ip 192.168.65.3 --highest-ip 192.168.65.254 --log-destination asl --udpv4-forwards 123:127.0.0.1:51637 --gc-compact-interval 1800
501 53336 53333   0 10:09AM ??         0:00.66 com.docker.driver.amd64-linux -addr fd:3 -debug
501 53339 53336   0 10:09AM ??         7:53.72 com.docker.hyperkit -A -u -F vms/0/hyperkit.pid -c 4 -m 5120M -s 0:0,hostbridge -s 31,lpc -s 1:0,virtio-vpnkit,path=vpnkit.eth.sock,uuid=447a9927-94d9-4358-8857-e62cf492e06e -U 9ed1818a-b94b-4788-8895-e08ce78e568b -s 2:0,ahci-hd,/Users/andreas.schafferer/Library/Containers/com.docker.docker/Data/vms/0/Docker.raw -s 3,virtio-sock,guest_cid=3,path=vms/0,guest_forwards=2376;1525 -s 4,ahci-cd,/Applications/Docker.app/Contents/Resources/linuxkit/docker-for-mac.iso -s 5,ahci-cd,vms/0/config.iso -s 6,ahci-cd,/Applications/Docker.app/Contents/Resources/linuxkit/docker.iso -s 7,virtio-rnd -s 8,virtio-9p,path=vpnkit.port.sock,tag=port -l com1,autopty=vms/0/tty,asl -f bootrom,/Applications/Docker.app/Contents/Resources/uefi/UEFI.fd,,
501 53442 52765   0 10:09AM ttys008    0:01.85 docker stats
501 54288 54086   0 10:24AM ttys011    0:00.00 grep docker