Redis self-healing cluster or alternative in cloud, Kubernetes or Swarm?

8/1/2017

Is it possible to deploy a self-healing and scaling Redis-like key-value store that I can run in Kubernetes (or Swarm or any other automated cloud env)?

The challenges I found with Redis:

  1. You need to create a cluster manually with redis-trib
  2. New nodes need to be added to the cluster explicitly
  3. Nodes need to be removed explicitly
  4. Nodes do not replicate data in their shards peer to peer, but rather use a master-slave model

The above means that the following scenarios all will fail. I have a simple 3-master and 3-slave cluster. "Master A" fails, leading Kubernetes to start a new "Master A" in <1 second:

  • new "Master A" has no knowledge of the cluster and will not join
  • "Slave A", which had copies of the data, now syncs from the new "Master A", and loses all data, defeating the purpose of slave replica
  • In completely new startup, "Master A" might start before "Master B", is initialized (assuming I even can automate the cluster init), before "Master B" is ready, and thus "Master B" never really joins the cluster

Questions:

  1. Is there a way to automate Redis cluster init and sync in a non-predictable, non-persistent storage without any human interaction?
  2. If not, is there an alternative that is a clean, in-memory key-value store (persistence is less important to me), that is self-healing and works peer-to-peer?

Consul/etcd/zookeeper all work 100% peer-to-peer and self-heal (which is great), but their performance (supposedly) is far below Redis in-memory KV. They aren't built, e.g. for looking up a session with each Web API request. This is partially due to non-sharding (100% copies), partially due to disk writes.

Kafka's model (although a message queue, not a KV store) works well too (but depends on zk underneath): partitions and replicas, but essentially self-healing. I talk to one broker, it tells me where my topic ("shard") lives, I get it.

Is there any way to get that full autonomy using Redis so I can deploy in kube/swarm/cloud, or an alternative that provides similar performance with the autonomous model?

-- deitch
consul
kubernetes
redis

1 Answer

8/2/2017

You can use the Kubernetes Redis Example which work with Redis Sentinel. In case the master fails, sentinel promotes a slave to the new master. The Replication Controller boots a new Slave Pod. Your application connects to sentinel and from that service you will get the ip of the new master.

Redis Sentinel https://redis.io/topics/sentinel

Kubernetes Example https://github.com/kubernetes/examples/tree/master/staging/storage/redis

-- KuLi
Source: StackOverflow