RabbitMQ, .NET Core and Google Kubernetes Engine (configuration)

2/24/2022

I'm trying to setup Rabitmq on GKE

my kubernetes rabbit service looks like

apiVersion: v1
kind: Service
metadata:
  name: rabbitmq
spec:
  type: LoadBalancer    
  ports:
  - protocol: TCP
    port: 5672
    targetPort: 5672
    name: amqp
  - protocol: TCP
    port: 4369
    targetPort: 4369
    name: discovery      
  - protocol: TCP
    port: 15672
    targetPort: 15672    
    name: management
  selector:
    app: rabbitmq

With external ip I am able to access the management UI just fine. but my C# code cannot connect to port 5672

34.152.xxx.xxx

is the external ip I get from the loadbalancer

ConnectionFactory factory = new ConnectionFactory();
factory.Uri = new Uri("amqp://username:password@34.152.xxx.xxx:5672/");
using(var connection = factory.CreateConnection())
using(var channel = connection.CreateModel())
{...}

I dont want to use service type as NodePort since i need to change the IP address if the node goes down.

What am I doing wrong? Cant seem to figure out.

Update Deployment

...
 containers:
      - name: rabbitmq
        image: rabbitmq:3.9-management
        ports:
        - containerPort: 4369
          name: discovery
        - containerPort: 5672
          name: amqp
        - containerPort: 15672
          name: management
...

output of kubectl describe pod

Name:         rabbitmq-rabbitmq-1
Namespace:    default
Priority:     0
Node:         gke-cluster-1-default-pool-46d75799-qzvs/10.162.15.206
Start Time:   Thu, 24 Feb 2022 23:05:31 +0000
Labels:       app.kubernetes.io/component=rabbitmq-server
              app.kubernetes.io/name=rabbitmq
              controller-revision-hash=rabbitmq-rabbitmq-f4d4975b
              statefulset.kubernetes.io/pod-name=rabbitmq-rabbitmq-1
Annotations:  <none>
Status:       Running
IP:           10.0.0.5
IPs:
  IP:           10.0.0.5
Controlled By:  StatefulSet/rabbitmq-rabbitmq
Init Containers:
  copy-rabbitmq-config:
    Container ID:  docker://75cdddb64bb5f6d4f70a66340e941581d45585d9d915fc5a79376c3aee352619
    Image:         gcr.io/cloud-marketplace/google/rabbitmq/debian9:3.9.13-20220220-155313
    Image ID:      docker-pullable://gcr.io/cloud-marketplace/google/rabbitmq/debian9@sha256:cda2b5cc143a59ec47e6d95ef2110a5531b1dc0f5bc42017a10b381042b8d629
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/bash
      -euc
      # Remove cached erlang cookie since we are always providing it,
      # that opens the way to recreate the application and access to existing data
      # as a new erlang will be regenerated again.
      echo ${RABBITMQ_ERLANG_COOKIE} > /var/lib/rabbitmq/.erlang.cookie
      chmod 600 /var/lib/rabbitmq/.erlang.cookie

      # Copy the mounted configuration to both places.
      cp /rabbitmqconfig/rabbitmq.conf /etc/rabbitmq/rabbitmq.conf
      # Change permission to allow to add more configurations via variables
      chown :999 /etc/rabbitmq/rabbitmq.conf
      chmod 660 /etc/rabbitmq/rabbitmq.conf
      cp /rabbitmqconfig/enabled_plugins /etc/rabbitmq/enabled_plugins

    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 24 Feb 2022 23:05:53 +0000
      Finished:     Thu, 24 Feb 2022 23:05:53 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      RABBITMQ_ERLANG_COOKIE:  <set to the key 'rabbitmq-erlang-cookie' in secret 'rabbitmq-rabbitmq-secret'>  Optional: false
    Mounts:
      /etc/rabbitmq from config (rw)
      /rabbitmqconfig from configmap (rw)
      /var/lib/rabbitmq from rabbitmq-rabbitmq-pvc (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from rabbitmq-rabbitmq-serviceaccount-661e-token-tl4hr (ro)
Containers:
  rabbitmq:
    Container ID:   docker://234f70edd6f5fd413efc86a29da728ff276b142b85052b8979d44a019d81b0ca
    Image:          gcr.io/cloud-marketplace/google/rabbitmq:3.9.13-20220220-155313
    Image ID:       docker-pullable://gcr.io/cloud-marketplace/google/rabbitmq@sha256:a8c26e7376eaa73dc4b45c9560ccca8ec0d3a87aa08863cc3b3ae733060ad32a
    Ports:          25672/TCP, 5672/TCP, 5671/TCP, 15692/TCP, 15672/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
    State:          Running
      Started:      Thu, 24 Feb 2022 23:06:13 +0000
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   100Mi
    Liveness:   exec [rabbitmqctl status] delay=60s timeout=30s period=10s #success=1 #failure=3
    Readiness:  exec [rabbitmqctl status] delay=20s timeout=30s period=10s #success=1 #failure=3
    Environment:
      JAVA_OPTS:              -Dlog4j2.formatMsgNoLookups=true -Dlog4j2.disable.jmx=true
      MY_POD_NAME:            rabbitmq-rabbitmq-1 (v1:metadata.name)
      RABBITMQ_USE_LONGNAME:  true
      RABBITMQ_NODENAME:      rabbit@$(MY_POD_NAME).rabbitmq-rabbitmq-discovery.default.svc.cluster.local
      K8S_SERVICE_NAME:       rabbitmq-rabbitmq-discovery
      K8S_HOSTNAME_SUFFIX:    .rabbitmq-rabbitmq-discovery.default.svc.cluster.local
      RABBITMQ_DEFAULT_USER:  rabbit
      RABBITMQ_DEFAULT_PASS:  <set to the key 'rabbitmq-pass' in secret 'rabbitmq-rabbitmq-secret'>  Optional: false
    Mounts:
      /etc/rabbitmq from config (rw)
      /var/lib/rabbitmq from rabbitmq-rabbitmq-pvc (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from rabbitmq-rabbitmq-serviceaccount-661e-token-tl4hr (ro)
  prometheus-to-sd:
    Container ID:  docker://45c7304a9e55f87d473e92a7371d8c3ec5eb5f9438b565c15e4855f2b97d9f2e
    Image:         gcr.io/cloud-marketplace/google/rabbitmq/prometheus-to-sd:3.9.13-20220220-155313
    Image ID:      docker-pullable://gcr.io/cloud-marketplace/google/rabbitmq/prometheus-to-sd@sha256:b254ffeee8c87dde0be0840782043e62d567a9238ba4fa5a662aaeb803099d94
    Port:          6060/TCP
    Host Port:     0/TCP
    Command:
      /monitor
      --stackdriver-prefix=custom.googleapis.com
      --source=rabbitmq:http://localhost:15692/metrics
      --pod-id=$(POD_NAME)
      --namespace-id=$(POD_NAMESPACE)
      --monitored-resource-type-prefix=k8s_
    State:          Running
      Started:      Thu, 24 Feb 2022 23:06:31 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      POD_NAME:       rabbitmq-rabbitmq-1 (v1:metadata.name)
      POD_NAMESPACE:  default (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from rabbitmq-rabbitmq-serviceaccount-661e-token-tl4hr (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  rabbitmq-rabbitmq-pvc:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  rabbitmq-rabbitmq-pvc-rabbitmq-rabbitmq-1
    ReadOnly:   false
  configmap:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rabbitmq-rabbitmq-config
    Optional:  false
  config:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  rabbitmq-rabbitmq-serviceaccount-661e-token-tl4hr:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rabbitmq-rabbitmq-serviceaccount-661e-token-tl4hr
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                From                     Message
  ----     ------                  ----               ----                     -------
  Warning  FailedScheduling        30m (x3 over 30m)  default-scheduler        0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.
  Normal   Scheduled               30m                default-scheduler        Successfully assigned default/rabbitmq-rabbitmq-1 to gke-cluster-1-default-pool-46d75799-qzvs
  Normal   SuccessfulAttachVolume  30m                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-a8302b5b-debc-4aa1-89bf-2d3295a201fe"
  Normal   Pulling                 30m                kubelet                  Pulling image "gcr.io/cloud-marketplace/google/rabbitmq/debian9:3.9.13-20220220-155313"
  Normal   Pulled                  30m                kubelet                  Successfully pulled image "gcr.io/cloud-marketplace/google/rabbitmq/debian9:3.9.13-20220220-155313" in 3.88917291s
  Normal   Created                 30m                kubelet                  Created container copy-rabbitmq-config
  Normal   Started                 30m                kubelet                  Started container copy-rabbitmq-config
  Normal   Pulling                 30m                kubelet                  Pulling image "gcr.io/cloud-marketplace/google/rabbitmq:3.9.13-20220220-155313"
  Normal   Pulled                  29m                kubelet                  Successfully pulled image "gcr.io/cloud-marketplace/google/rabbitmq:3.9.13-20220220-155313" in 16.874411606s
  Normal   Created                 29m                kubelet                  Created container rabbitmq
  Normal   Started                 29m                kubelet                  Started container rabbitmq
  Normal   Pulling                 29m                kubelet                  Pulling image "gcr.io/cloud-marketplace/google/rabbitmq/prometheus-to-sd:3.9.13-20220220-155313"
  Normal   Pulled                  29m                kubelet                  Successfully pulled image "gcr.io/cloud-marketplace/google/rabbitmq/prometheus-to-sd:3.9.13-20220220-155313" in 1.601774667s
  Normal   Created                 29m                kubelet                  Created container prometheus-to-sd
  Normal   Started                 29m                kubelet                  Started container prometheus-to-sd

New Update

With service as NodePort I'm able to connect to the amqp port but now the some times it connects and sometimes it doesnt. If i run my sample test it connects -> publishes the message and terminates. Then again when i run it doesnt connect. When I try after waiting a minute it connects again.

Seems like whenever the connection is terminated rabbitmq needs time to listen again.

I haven't been able to connect at all with service type loadbalancer yet.

-- Tejas Ghutukade
gke-networking
google-kubernetes-engine
kubernetes
kubernetes-pod
rabbitmq

0 Answers