Redis Sentinel doesn't auto discover new slave instances

12/27/2021

I've deployed the redis helm chart on k8s with Sentinel enabled.

I've set up the Master-Replicas with Sentinel topology, it means one master and two slaves. Each pod is running both the redis and sentinel container successfully:

NAME             READY   STATUS    RESTARTS   AGE     IP             NODE
my-redis-pod-0   2/2     Running   0          5d22h   10.244.0.173   node-pool-u
my-redis-pod-1   2/2     Running   0          5d22h   10.244.1.96    node-pool-j
my-redis-pod-2   2/2     Running   0          3d23h   10.244.1.145   node-pool-e

Now, I've a python script that connects to redis and discovers the master by passing it the pod's ip.

sentinel = Sentinel([('10.244.0.173', 26379),
                     ('10.244.1.96',26379),
                     ('10.244.1.145',26379)],
                  sentinel_kwargs={'password': 'redispswd'})

host, port = sentinel.discover_master('mymaster')
redis_client = StrictRedis(
            host=host,
            port=port,
            password='redispswd')

Let's suposse the master node is on my-redis-pod-0, when I do kubectl delete pod to simulate a problem that leads me to loss the pod, Sentinel will promote one of the others slaves to master and kubernetes will give me a new pod with redis and sentinel.

NAME             READY   STATUS    RESTARTS   AGE     IP             NODE
my-redis-pod-0   2/2     Running   0          3m      10.244.0.27    node-pool-u
my-redis-pod-1   2/2     Running   0          5d22h   10.244.1.96    node-pool-j
my-redis-pod-2   2/2     Running   0          3d23h   10.244.1.145   node-pool-e

The question is, how can I do to tell Sentinel to add this new ip to the list automatically (without code changes)?

Thanks!

-- Martín C.
helm3
kubernetes
redis
redis-sentinel

1 Answer

12/28/2021

Instead of using IPs, you may use the dns entries for a headless service.

A headless service is created by explicitly specifying

ClusterIP: None

Then you will be able to use the dns entries as under, where redis-0 will be the master

#syntax
pod_name.service_name.namespace.svc.cluster.local

#Example
redis-0.redis.redis.svc.cluster.local
redis-1.redis.redis.svc.cluster.local
redis-2.redis.redis.svc.cluster.local

References: https://stackoverflow.com/questions/52707840/what-is-a-headless-service-what-does-it-do-accomplish-and-what-are-some-legiti

https://www.containiq.com/post/deploy-redis-cluster-on-kubernetes

-- Rakesh Gupta
Source: StackOverflow