How can I limit the usage of swap memory in k8s?

3/11/2020

I know k8s does not support swap by default.

But since I have a rather static deployment of pods in k8s (3 nodes, each with a solr + zookeeper pod), so I decided to set up 2G swap on all my worker nodes anyway and allow the use of swap by starting kubelet with --fail-swap-on=false

Now I have got the cluster up & running, and it seemed to be running okay.

However, I found that my java processes are using a lot of swap, I am worried that this might affect the performance.

  • RSS: 317160 Kb
  • SWAP: 273232 Kb

My question is that is there a way to limit the use of swap memory usage by containers?

I was thinking about setting --memory-swap params of Docker

Currently, based on Docker inspect my container has no limit on swap usage ( "MemorySwap": -1 )

sudo docker inspect 482d70f73c7c | grep Memory
            "Memory": 671088640,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": -1,
            "MemorySwappiness": null,

But I just couldn't find this param exposed in k8s.

My vm-related settings

vm.overcommit_kbytes = 0
vm.overcommit_memory = 1
vm.overcommit_ratio = 50
vm.swappiness = 20
vm.vfs_cache_pressure = 1000

p.s. Will the limit on pod memory also limit the swap usage?

Thank you all for reading the post!

Notes

This is how I got the RSS & SWAP of my Java process

Run top

27137 8983 20 0 3624720 317160 0 S 3.0 32.0 4:06.74 java

Run

find /proc -maxdepth 2 -path "/proc/[0-9]*/status" -readable -exec awk -v FS=":" '{process[$1]=$2;sub(/^[ \t]+/,"",process[$1]);} END {if(process["VmSwap"] && process["VmSwap"] != "0 kB") printf "%10s %-30s %20s\n",process["Pid"],process["Name"],process["VmSwap"]}' '{}' \; | awk '{print $(NF-1),$0}' | sort -h | cut -d " " -f2-

27137 java 273232 kB

-- John the Traveler
docker
kubernetes

1 Answer

3/11/2020

And now you know why the kubelet literally refuses to start by default if there swap active. This is not exposed anywhere in CRI and won’t be.

-- coderanger
Source: StackOverflow