I have generated a list of keyspaces along with their size and ttl (715 keyspaces) and trying to analyze which keyspace is consuming huge memory. I can see that total memory occupied by the keyspaces is 1.2 MB but my Redis PVC is filled up to 13 GB. So, what is occupying so much memory in Redis container ?
Redis Version: 6.0.9 Image: bitnami/redis:6.0.9-debian-10-r0
Script I used to generate the list:
#!/bin/bash
NAMESPACE=$1
SECRET=$2
POD=$3
CONTAINER=$4
PASSWORD_REDIS=$(kubectl get secrets -n $NAMESPACE $SECRET -o jsonpath='{.data.secret'} | base64 -d )
kubectl exec -it -n $NAMESPACE $POD -c $CONTAINER -- redis-cli -a ${PASSWORD_REDIS} KEYS '*' | tail -n +2 > keylist.csv
keylist.csv
declare -a KEYS;
arraylen=0;
i=0;
j=0;
test2(){
arraylen=${#KEYS[@]}
echo "kubectl exec -i -n ${NAMESPACE} ${POD} -c ${CONTAINER} -- redis-cli -h 127.0.0.1 -a ${PASSWORD_REDIS} MEMORY USAGE ${KEYS[j]}" > keys.sh
chmod +x keys.sh
./keys.sh > size.txt 2> /dev/null
echo "kubectl exec -i -n ${NAMESPACE} ${POD} -c ${CONTAINER} -- redis-cli -h 127.0.0.1 -a ${PASSWORD_REDIS} TTL ${KEYS[j]}" > ttl.sh
chmod +x ttl.sh
./ttl.sh > ttl.txt 2> /dev/null
SIZE=$(cat size.txt)
TTL=$(cat ttl.txt)
((j=$j+1))
echo "$j,${KEYS[j]},${SIZE},${TTL}" >> output.csv
if [ $j -ge $arraylen ]; then exit
fi
test2
}
while IFS=') ' read -r slno line; do
LINE=${line};
KEYS+=([i]=$LINE)
i=$[$i+1]
done < keylist.csv
test2
Output looks like this (3rd column is size in bytes and 4th column is TTL in ms):
Documentation of what is stored in GitLab's redis cache can be found here and here.
It changes over time and depends on the version of GitLab you are using.
It's also worth mentioning that GitLab has built a tool, redis-keyspace-analyzer for analyzing the keyspace of GitLab redis instances. You can use this to find the keys taking up the most space. (see also redis-bigkeys and its infra notes). These tools are built by gitlab.com engineers for use with gitlab.com but should be adaptable to self-hosted gitlab instances.