K
Q

How to reconnect Redis cluster nodes?

2/15/2019

I have a Redis cluster of 6 nodes, running in my Kubernetes cluster as a stateful set. Since it was for testing and not yet on production, all of the Redis nodes were on the same machine. Of course, the machine failed, and all of Redis' nodes crashed immediately. When the machine was back alive the pods were recreated and was given different cluster ips, therefore they couldn't re-connect with each other.

I need to find a solution for a disaster case such as this. Let's assume all the nodes were reassigned with different ips, how can I configure the nodes to get to other ips?

The slaves are easy to reset with the CLUSTER RESET command, but the masters contain slots and data that shouldn't be deleted.

Should I manually re-write nodes.conf? I'm afraid this will make it even worse? I there a known method to deal with it?

Thanks!

-- Yuval
kubernetes
kubernetes-statefulset
redis
redis-cluster

Similar Questions

How to customise config.toml on Kubernetes?
RabbitMQ management - 404 when loading a queue or exchange
Kubernetes ingress nginx grafana dashboard prometheus data was empty
JupyterHub Hub Pod stuck in pending - pod has unbound immediate PersistentVolumeClaims

2 Answers

2/15/2019

Found a solution:

The first step is to change the current pod ip in nodes.conf when the pod is starting. You can achieve that with this script

#!/bin/sh
    CLUSTER_CONFIG="/data/nodes.conf"
    if [ -f ${CLUSTER_CONFIG} ]; then
      if [ -z "${POD_IP}" ]; then
        echo "Unable to determine Pod IP address!"
        exit 1
      fi
      echo "Updating my IP to ${POD_IP} in ${CLUSTER_CONFIG}"
      sed -i.bak -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/" ${CLUSTER_CONFIG}
    fi
    exec "$@"

You should start any pod by calling this script and passing to it the original redis-server start command.

Now every pod in the cluster has the correct IP of itself set.

  1. Make sure the cluster's pods are stable and don't crash.
  2. Manually edit nodes.conf in one of the pods. Set the correct IPs instead of the deprecated ones.
  3. Restart the pod you've edit with redis-cli shutdown. Kubernetes will set up a new pod for it. The new pod's IP will be set by the script I added above.
-- Yuval
Source: StackOverflow
2/15/2019

In my opinion you shouldn't rely on Pods' internal IP address at all, when referencing your Redis cluster in any place within your application. Pods are mortal, this means they are designed to crash. So when node dies, they are destroyed too. When node resurrects, PODs are recreated with new IP addresses.

The proper way to target your PODs would be via their DNS names (as explained here), if you created your Redis cluster as a Stateful application.

-- Nepomucen
Source: StackOverflow