Elixir - Using distributed Cachex with Kubernetes. How to set the :nodes config

2/25/2019

So I want to use Cachex to share temporary data across the cluster. Cachex takes a :nodes list in the config file. That worked great in testing, as I could just hard code [:"a@My-Laptop", :"b@My-Laptop"] and it worked, but with Kubernetes the names are dynamic. How would I set these?

-- Peter R
elixir
kubernetes

2 Answers

2/26/2019

I think this should work:

In your pod configuration set an ENV variable for the pod's IP

apiVersion: v1
kind: Pod
spec:
  containers:
    - name: my-container
      env:
        - name: MY_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP

Then in your elixir app you can now use myapp@${MY_POD_IP}

Here is the documentation for exposing node/cluster information to your app https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/

I also found this blog post https://medium.com/polyscribe/a-complete-guide-to-deploying-elixir-phoenix-applications-on-kubernetes-part-5-clustering-6c30fcd35ce1 that explains a bit more, what you are trying to do.

-- acrogenesis
Source: StackOverflow

3/29/2019

I looked at this for my day job a while back and Cachex doesn't have a solution for dynamic node membership. It expects you to have all of your node names predefined and static. You can do this in Kubernetes with stateful sets, but you'll throw away all future possibility of autoscaling if you want to keep your cache distributed across all nodes.

A better solution IMO is Nebulex. Nebulex is not concerned about managing the connection to your nodes or their membership. It is only concerned with keeping a cache distributed across [Node.self() | Node.list()] (all of the nodes in your cluster). It all runs over PG2 which is a distributed pubsub implementation. It has the ability to transfer data as nodes join or leave and will shard it so that you have duplicate data across your nodes. It even allows you to match on node names to store your cached data.

To manage connecting your nodes to each other in a dynamic way, there's a library called libcluster that actually has Kubernetes support built in. Libcluster will either use Kubernetes DNS or you can create a headless service that will serve the IP addresses of all your elixir pods. Libcluster will ping this service on a polling interval and try to join the erlang cluster.

When you have both set up, libcluster will manage keeping your nodes in a cluster and Nebulex will manage keeping your data synced across all of them. It's pretty cool and we haven't had any problems with it yet.

I'm not going to write any source here because the documentation for all of these things is really well written. Let me know if you have any questions though.

-- Mike Quinlan
Source: StackOverflow