K
Q

The memory of cgroup rss is much higher than the summary of the memory usage of all processes in the docker container

3/25/2021

I hava a Redis runing in a container . Inside the container cgroup rss show using about 1283MB memory.

The kmem memory usage is 30.75MB.

The summary of the memory usage of all processes in the docker container is 883MB.

How can i figure out the "disappeared memory "(1296-883-30=383MB).The "disappeared memory" will growing with the time pass.Flinally the container will be oom killed .

environmet info is

redis version:4.0.1

docker version:18.09.9

k8s version:1.13

the memory usage is 1283MB 

root@redis-m-s-rbac-0:/opt#cat /sys/fs/cgroup/memory/memory.usage_in_bytes 
1346289664  >>>> 1283.921875 MB

the kmem memory usage is 30.75MB

root@redis-m-s-rbac-0:/opt#cat /sys/fs/cgroup/memory/memory.kmem.usage_in_bytes 
32194560  >>> 30.703125 MB
root@redis-m-s-rbac-0:/opt#cat /sys/fs/cgroup/memory/memory.stat 
cache 3358720
rss 1359073280 >>>  1296.11328125 MB
rss_huge 515899392
shmem 0
mapped_file 405504
dirty 0
writeback 0
swap 0
pgpgin 11355630
pgpgout 11148885
pgfault 25710366
pgmajfault 0
inactive_anon 0
active_anon 1359245312
inactive_file 2351104
active_file 1966080
unevictable 0
hierarchical_memory_limit 4294967296
hierarchical_memsw_limit 4294967296
total_cache 3358720
total_rss 1359073280
total_rss_huge 515899392
total_shmem 0
total_mapped_file 405504
total_dirty 0
total_writeback 0
total_swap 0
total_pgpgin 11355630
total_pgpgout 11148885
total_pgfault 25710366
total_pgmajfault 0
total_inactive_anon 0
total_active_anon 1359245312
total_inactive_file 2351104
total_active_file 1966080
total_unevictable 0

the summary of the memory usage of all processes in the docker container is 883MB 

root@redis-m-s-rbac-0:/opt#ps aux | awk '{sum+=$6} END {print sum / 1024}'
883.609
-- nobb
cgroups
docker
kubernetes
memory
redis

Similar Questions

Helm chart versions for CI/CD
Unable to scrape metrics from pods
helm yaml error - found unexpected ':'
gcloud Issue with Persistent Volume running OracleDB Docker image (Permission denied)
Kafka Chart on Kubernetes: Simple test string produce + consume

1 Answer

3/25/2021

This is happening because usage_in_bytes does not show exact value of memory and swap usage. The memory.usage_in_bytes show current memory(RSS+Cache) usage.

5.5 usage_in_bytes For efficiency, as other kernel components, memory cgroup uses some optimization to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the method and doesn't show 'exact' value of memory (and swap) usage, it's a fuzz value for efficient access. (Of course, when necessary, it's synchronized.) If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP) value in memory.stat(see 5.2).

Reference: https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt

-- acid_fuji
Source: StackOverflow