K
Q

redis memory usage exceeds k8s resource limit

9/28/2020

A redis instance is deployed in k8s cluster, memory limit is 1G:

resources:
  limits:
    memory: 1Gi

And also set redis maxmemory to 1G, actual memory usage is ~550M:

info memory
# Memory
used_memory:577791152
used_memory_human:551.02M
used_memory_rss:611942400
used_memory_rss_human:583.59M
used_memory_peak:578877888
used_memory_peak_human:552.06M
used_memory_peak_perc:99.81%
used_memory_overhead:32108388
used_memory_startup:786608
used_memory_dataset:545682764
used_memory_dataset_perc:94.57%
total_system_memory:67555897344
total_system_memory_human:62.92G
used_memory_lua:41984
used_memory_lua_human:41.00K
maxmemory:1000000000
maxmemory_human:953.67M
maxmemory_policy:noeviction
mem_fragmentation_ratio:1.06
mem_allocator:jemalloc-4.0.3
active_defrag_running:0
lazyfree_pending_objects:0

I observed that the redis container memory usage exceeds 1G from prometheus (metric sum by(container_name) (container_memory_working_set_bytes{namespace="$namespace",container_name!="",container_name=~"$container",container_name!="POD",pod_name="$pod"})):

enter image description here

So here is the question:

  1. How to free up the memory?

  2. what's the relationship between container_memory_working_set_bytes and redis reported used_memory_rss used_memory?

    1: https://i.stack.imgur.com/JkUB4.png

-- chanjarster
kubernetes
prometheus
redis

Similar Questions

Session Affinity in kubernetes based on Client session not on Client IP
How to use K8S env variable in python
How to have one RabbitMQ cluster consisting of 2 Kubernetes statefulsets?
List of all reasons for Container States in kubernetes
Kubernetes bind address

1 Answer

9/28/2020

This is a problem relating to a k8s prometheus issue, prometheus actually got a doubled result:

container_cpu_usage_seconds_total{container="redis-server",container_name="redis-server",cpu="total",endpoint="https-metrics",id="/kubepods/pod149c0943-bc55-4e5d-ac9d-59fedca56138/1e2f5a63ddba49cb13465383e0de66e50ce9f084b52975f9bea8d1d77b966b3c",image="sha256:8a74675d2c029efb8c574f6097f2494d0e6acb7427aeea874227be48d96a3fb2",instance="192.168.116.119:10250",job="expose-kubelets-metrics",name="k8s_redis-server_redis-server-0_user-data-service-test_149c0943-bc55-4e5d-ac9d-59fedca56138_0",namespace="user-data-service-test",node="k8sworker03",pod="redis-server-0",pod_name="redis-server-0",service="expose-kubelets-metrics"}

container_cpu_usage_seconds_total{container="redis-server",container_name="redis-server",endpoint="https-metrics",instance="192.168.116.119:10250",job="expose-kubelets-metrics",namespace="user-data-service-test",node="k8sworker03",pod="redis-server-0",pod_name="redis-server-0",service="expose-kubelets-metrics"}

So the result is doubbled.

-- chanjarster
Source: StackOverflow