K
Q

Unable to start RabbitMQ image in AKS Cluster

6/5/2019

I am attempting to start a RabbitMQ image in my AKS cluster. The VMs comprising the cluster are on a private VNET and have firewall rules in place.

What needs to be allowed through the firewall isn't clear (or if it's even the problem).

Here's the output when the pod starts:

BOOT FAILED

Config file generation failed: Failed to create dirty io scheduler thread 6, error = 11

Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...Segmentation fault (core dumped)

{"init terminating in do_boot",generate_config_file} init terminating in do_boot (generate_config_file)

Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done

I have attached persistent volumes to /var/log and /var/lib/rabbitmq but there's no log files or anything else that aids in debugging this issue. Schema, lost+found, and other rabbitmq folders and files are created, so it's reading/writing fine.

Here's the YAML I'm using to create the pod:

   apiVersion: extensions/v1beta1
   kind: Deployment
   metadata:
     name: mayan-broker
   spec:
     replicas: 1
     template:
      metadata:
       labels:
         app: mayan-broker
      spec:
        containers:                           
         - name: mayan-broker
           image: rabbitmq:3
           volumeMounts:
           - name: broker-storage
             mountPath: /var/lib/rabbitmq
           - name: broker-logging
             mountPath: /var/log/rabbitmq
           ports:
             - containerPort: 5672
           env:
               -  name: RABBITMQ_DEFAULT_USER
                  value: mayan
               -  name: RABBITMQ_DEFAULT_PASS
                  value: mayan
               -  name: RABBITMQ_DEFAULT_VHOST
                  value: mayan      
        volumes:
         - name: broker-storage
           persistentVolumeClaim:
             claimName: rabbit-claim    
         - name: broker-logging
           persistentVolumeClaim:
             claimName: logging-claim

YAML without volumes and mounts per request, yielding the same result:

   apiVersion: extensions/v1beta1
   kind: Deployment
   metadata:
     name: mayan-broker
   spec:
     replicas: 1
     template:
      metadata:
       labels:
         app: mayan-broker
      spec:
        containers:                           
         - name: mayan-broker
           image: rabbitmq:3
           ports:
             - containerPort: 5672
           env:
               -  name: RABBITMQ_DEFAULT_USER
                  value: mayan
               -  name: RABBITMQ_DEFAULT_PASS
                  value: mayan
               -  name: RABBITMQ_DEFAULT_VHOST
                  value: MAYAN
-- wvisaacs
azure-aks
kubernetes
rabbitmq

Similar Questions

Unable to List Available Docker Version
GRPC Load Balancing with Envoy
Is there a way to delete pods automatically through YAML after they have status 'Completed'?
Why can't I build and then run my container locally
Work on docker several users same file frome different places
How to inject evnironment variables to driver pod when using spark-on-k8s?
Kyma restart issue in local
What registry I have to pass when installing kamel on kubernetes?

1 Answer

8/6/2019

I had the same problem with AKS (I'm starting to think it's an AKS thing).

Basically AKS limits the number of thread a pod can use, and rabbitmq (and everything Erlang in general) uses a lot of threads.

You can use env vars in your yaml to reduce the number of threads, like in my config:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
spec:
  serviceName: "rabbitmq"
  replicas: 1
  selector:
    matchLabels:
      app: rabbitmq
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      containers:
      - name: rabbitmq
        image: rabbitmq:3.7-management
        env:
            # this needs to be there because AKS (as of 1.14.3)
            # limits the number of thread a pod can use
            - name: RABBITMQ_IO_THREAD_POOL_SIZE
              value: "30"
        ports:
        - containerPort: 5672
          name: amqp
        resources:
          limits:
            memory: 4Gi
          requests:
            cpu: "1"
            memory: 1Gi

I'm using a statefulsets, but the fix is the same for a deployment.

-- Vincent Cloutier
Source: StackOverflow