I know k8s does not support swap by default.
But since I have a rather static deployment of pods in k8s (3 nodes, each with a solr + zookeeper pod), so I decided to set up 2G swap on all my worker nodes anyway and allow the use of swap by starting kubelet with --fail-swap-on=false
Now I have got the cluster up & running, and it seemed to be running okay.
However, I found that my java processes are using a lot of swap, I am worried that this might affect the performance.
My question is that is there a way to limit the use of swap memory usage by containers?
I was thinking about setting --memory-swap
params of Docker
Currently, based on Docker inspect my container has no limit on swap usage ( "MemorySwap": -1
)
sudo docker inspect 482d70f73c7c | grep Memory
"Memory": 671088640,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": -1,
"MemorySwappiness": null,
But I just couldn't find this param exposed in k8s.
My vm-related settings
vm.overcommit_kbytes = 0
vm.overcommit_memory = 1
vm.overcommit_ratio = 50
vm.swappiness = 20
vm.vfs_cache_pressure = 1000
p.s. Will the limit on pod memory also limit the swap usage?
Thank you all for reading the post!
This is how I got the RSS & SWAP of my Java process
Run top
27137 8983 20 0 3624720 317160 0 S 3.0 32.0 4:06.74 java
Run
find /proc -maxdepth 2 -path "/proc/[0-9]*/status" -readable -exec awk -v FS=":" '{process[$1]=$2;sub(/^[ \t]+/,"",process[$1]);} END {if(process["VmSwap"] && process["VmSwap"] != "0 kB") printf "%10s %-30s %20s\n",process["Pid"],process["Name"],process["VmSwap"]}' '{}' \; | awk '{print $(NF-1),$0}' | sort -h | cut -d " " -f2-
27137 java 273232 kB
And now you know why the kubelet literally refuses to start by default if there swap active. This is not exposed anywhere in CRI and won’t be.