K
Q

Changing hostname breaks Rabbitmq when running on Kubernetes

4/26/2016

I'm trying to run Rabbitmq using Kubernetes on AWS. I'm using the official Rabbitmq docker container. Each time the pod restarts the rabbitmq container gets a new hostname. I've setup a service (of type LoadBalancer) for the pod with a resolvable DNS name.

But when I use an EBS to make the rabbit config/messsage/queues persistent between restarts it breaks with:

exception exit: {{failed_to_cluster_with,
                     ['rabbitmq@rabbitmq-deployment-2901855891-nord3'],
                     "Mnesia could not connect to any nodes."},
                 {rabbit,start,[normal,[]]}}
  in function  application_master:init/4 (application_master.erl, line 134)

rabbitmq-deployment-2901855891-nord3 is the previous hostname rabbitmq container. It is almost like Mnesia saved the old hostname :-/

The container's info looks like this:

              Starting broker...
=INFO REPORT==== 25-Apr-2016::12:42:42 ===
node           : rabbitmq@rabbitmq-deployment-2770204827-cboj8
home dir       : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash    : XXXXXXXXXXXXXXXX
log            : tty
sasl log       : tty
database dir   : /var/lib/rabbitmq/mnesia/rabbitmq

I'm only able to set the first part of the node name to rabbitmq using the RABBITMQ_NODENAME environment variable.

Setting RABBITMQ_NODENAME to a resolvable DNS name breaks with:

Can't set short node name!\nPlease check your configuration\n"

Setting RABBITMQ_USE_LONGNAME to true breaks with:

Can't set long node name!\nPlease check your configuration\n"

Update:

  • Setting RABBITMQ_NODENAME to rabbitmq@localhost works but that negates any possibility to cluster instances.

              Starting broker...
=INFO REPORT==== 26-Apr-2016::11:53:19 ===
node           : rabbitmq@localhost
home dir       : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash    : 9WtXr5XgK4KXE/soTc6Lag==
log            : tty
sasl log       : tty
database dir   : /var/lib/rabbitmq/mnesia/rabbitmq@localhost

  • Setting RABBITMQ_NODENAME to the service name, in this case rabbitmq-service like so rabbitmq@rabbitmq-service also works since kubernetes service names are internally resolvable via DNS.

              Starting broker...
=INFO REPORT==== 26-Apr-2016::11:53:19 ===
node           : rabbitmq@rabbitmq-service
home dir       : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash    : 9WtXr5XgK4KXE/soTc6Lag==
log            : tty
sasl log       : tty
database dir   : /var/lib/rabbitmq/mnesia/rabbitmq@rabbitmq-service

Is this the right way though? Will I still be able to cluster multiple instances if the node names are the same?

-- Pieter van der Merwe
amazon-ec2
dns
hostname
kubernetes
rabbitmq

Similar Questions

kubectl run command is failing with a connection refused error
Prevent back-off in kubernetes crash loop
Kubernetes when to use secrets instead of configmap?
Pod Containers Keeps on Restarting

2 Answers

4/26/2016

The idea is to use a different 'service' and 'deployment' for each of the node you want to create.

As you said, you have to create a custom NODENAME for each i.e:

RABBITMQ_NODENAME=rabbit@rabbitmq-1

Also rabbitmq-1,rabbitmq-2,rabbitmq-3 have to be resolved from each nodes. For that you can use kubedns. The /etc/resolv.conf will look like:

search rmq.svc.cluster.local

and /etc/hosts must contains:

127.0.0.1 rabbitmq-1  # or rabbitmq-2 on node 2...

The services are here to create a stable network identity for each nodes

rabbitmq-1.svc.cluster.local
rabbitmq-2.svc.cluster.local
rabbitmq-3.svc.cluster.local

The different deployments resources will allow you to mount a different volume on each node.

I'm working on a deployment tool to simplify those actions: I've done a demo on how I scale and deploy rabbitmq from 1 to 3 nodes on kubernetes: https://asciinema.org/a/2ktj7kr2d2m3w25xrpz7mjkbu?speed=1.5

More generally, the complexity your facing to deploy a clustered application is addressed in the 'petset proposal': https://github.com/kubernetes/kubernetes/pull/18016

-- ant31
Source: StackOverflow
4/28/2016

In addition to the first reply by @ant31:

Kubernetes now allows to setup a hostname, e.g. in yaml:

template:
  metadata:
    annotations:
      "pod.beta.kubernetes.io/hostname": rabbit-rc1

See https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns # A Records and hostname Based on Pod Annotations - A Beta Feature in Kubernetes v1.2

It seems that the whole configuration alive multiple restarts or re-schedules. I've not setup a cluster however I'm going to follow the tutorial for mongodb, see https://www.mongodb.com/blog/post/running-mongodb-as-a-microservice-with-docker-and-kubernetes

The approach will be probably almost same from kubernetes point of view.

-- Martin Podval
Source: StackOverflow