I'm trying to setup Rabitmq on GKE
my kubernetes rabbit service looks like
apiVersion: v1
kind: Service
metadata:
name: rabbitmq
spec:
type: LoadBalancer
ports:
- protocol: TCP
port: 5672
targetPort: 5672
name: amqp
- protocol: TCP
port: 4369
targetPort: 4369
name: discovery
- protocol: TCP
port: 15672
targetPort: 15672
name: management
selector:
app: rabbitmq
With external ip I am able to access the management UI just fine. but my C# code cannot connect to port 5672
34.152.xxx.xxx
is the external ip I get from the loadbalancer
ConnectionFactory factory = new ConnectionFactory();
factory.Uri = new Uri("amqp://username:password@34.152.xxx.xxx:5672/");
using(var connection = factory.CreateConnection())
using(var channel = connection.CreateModel())
{...}
I dont want to use service type as NodePort since i need to change the IP address if the node goes down.
What am I doing wrong? Cant seem to figure out.
Update Deployment
...
containers:
- name: rabbitmq
image: rabbitmq:3.9-management
ports:
- containerPort: 4369
name: discovery
- containerPort: 5672
name: amqp
- containerPort: 15672
name: management
...
output of kubectl describe pod
Name: rabbitmq-rabbitmq-1
Namespace: default
Priority: 0
Node: gke-cluster-1-default-pool-46d75799-qzvs/10.162.15.206
Start Time: Thu, 24 Feb 2022 23:05:31 +0000
Labels: app.kubernetes.io/component=rabbitmq-server
app.kubernetes.io/name=rabbitmq
controller-revision-hash=rabbitmq-rabbitmq-f4d4975b
statefulset.kubernetes.io/pod-name=rabbitmq-rabbitmq-1
Annotations: <none>
Status: Running
IP: 10.0.0.5
IPs:
IP: 10.0.0.5
Controlled By: StatefulSet/rabbitmq-rabbitmq
Init Containers:
copy-rabbitmq-config:
Container ID: docker://75cdddb64bb5f6d4f70a66340e941581d45585d9d915fc5a79376c3aee352619
Image: gcr.io/cloud-marketplace/google/rabbitmq/debian9:3.9.13-20220220-155313
Image ID: docker-pullable://gcr.io/cloud-marketplace/google/rabbitmq/debian9@sha256:cda2b5cc143a59ec47e6d95ef2110a5531b1dc0f5bc42017a10b381042b8d629
Port: <none>
Host Port: <none>
Command:
/bin/bash
-euc
# Remove cached erlang cookie since we are always providing it,
# that opens the way to recreate the application and access to existing data
# as a new erlang will be regenerated again.
echo ${RABBITMQ_ERLANG_COOKIE} > /var/lib/rabbitmq/.erlang.cookie
chmod 600 /var/lib/rabbitmq/.erlang.cookie
# Copy the mounted configuration to both places.
cp /rabbitmqconfig/rabbitmq.conf /etc/rabbitmq/rabbitmq.conf
# Change permission to allow to add more configurations via variables
chown :999 /etc/rabbitmq/rabbitmq.conf
chmod 660 /etc/rabbitmq/rabbitmq.conf
cp /rabbitmqconfig/enabled_plugins /etc/rabbitmq/enabled_plugins
State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 24 Feb 2022 23:05:53 +0000
Finished: Thu, 24 Feb 2022 23:05:53 +0000
Ready: True
Restart Count: 0
Environment:
RABBITMQ_ERLANG_COOKIE: <set to the key 'rabbitmq-erlang-cookie' in secret 'rabbitmq-rabbitmq-secret'> Optional: false
Mounts:
/etc/rabbitmq from config (rw)
/rabbitmqconfig from configmap (rw)
/var/lib/rabbitmq from rabbitmq-rabbitmq-pvc (rw)
/var/run/secrets/kubernetes.io/serviceaccount from rabbitmq-rabbitmq-serviceaccount-661e-token-tl4hr (ro)
Containers:
rabbitmq:
Container ID: docker://234f70edd6f5fd413efc86a29da728ff276b142b85052b8979d44a019d81b0ca
Image: gcr.io/cloud-marketplace/google/rabbitmq:3.9.13-20220220-155313
Image ID: docker-pullable://gcr.io/cloud-marketplace/google/rabbitmq@sha256:a8c26e7376eaa73dc4b45c9560ccca8ec0d3a87aa08863cc3b3ae733060ad32a
Ports: 25672/TCP, 5672/TCP, 5671/TCP, 15692/TCP, 15672/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
State: Running
Started: Thu, 24 Feb 2022 23:06:13 +0000
Ready: True
Restart Count: 0
Requests:
cpu: 100m
memory: 100Mi
Liveness: exec [rabbitmqctl status] delay=60s timeout=30s period=10s #success=1 #failure=3
Readiness: exec [rabbitmqctl status] delay=20s timeout=30s period=10s #success=1 #failure=3
Environment:
JAVA_OPTS: -Dlog4j2.formatMsgNoLookups=true -Dlog4j2.disable.jmx=true
MY_POD_NAME: rabbitmq-rabbitmq-1 (v1:metadata.name)
RABBITMQ_USE_LONGNAME: true
RABBITMQ_NODENAME: rabbit@$(MY_POD_NAME).rabbitmq-rabbitmq-discovery.default.svc.cluster.local
K8S_SERVICE_NAME: rabbitmq-rabbitmq-discovery
K8S_HOSTNAME_SUFFIX: .rabbitmq-rabbitmq-discovery.default.svc.cluster.local
RABBITMQ_DEFAULT_USER: rabbit
RABBITMQ_DEFAULT_PASS: <set to the key 'rabbitmq-pass' in secret 'rabbitmq-rabbitmq-secret'> Optional: false
Mounts:
/etc/rabbitmq from config (rw)
/var/lib/rabbitmq from rabbitmq-rabbitmq-pvc (rw)
/var/run/secrets/kubernetes.io/serviceaccount from rabbitmq-rabbitmq-serviceaccount-661e-token-tl4hr (ro)
prometheus-to-sd:
Container ID: docker://45c7304a9e55f87d473e92a7371d8c3ec5eb5f9438b565c15e4855f2b97d9f2e
Image: gcr.io/cloud-marketplace/google/rabbitmq/prometheus-to-sd:3.9.13-20220220-155313
Image ID: docker-pullable://gcr.io/cloud-marketplace/google/rabbitmq/prometheus-to-sd@sha256:b254ffeee8c87dde0be0840782043e62d567a9238ba4fa5a662aaeb803099d94
Port: 6060/TCP
Host Port: 0/TCP
Command:
/monitor
--stackdriver-prefix=custom.googleapis.com
--source=rabbitmq:http://localhost:15692/metrics
--pod-id=$(POD_NAME)
--namespace-id=$(POD_NAMESPACE)
--monitored-resource-type-prefix=k8s_
State: Running
Started: Thu, 24 Feb 2022 23:06:31 +0000
Ready: True
Restart Count: 0
Environment:
POD_NAME: rabbitmq-rabbitmq-1 (v1:metadata.name)
POD_NAMESPACE: default (v1:metadata.namespace)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from rabbitmq-rabbitmq-serviceaccount-661e-token-tl4hr (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
rabbitmq-rabbitmq-pvc:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: rabbitmq-rabbitmq-pvc-rabbitmq-rabbitmq-1
ReadOnly: false
configmap:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: rabbitmq-rabbitmq-config
Optional: false
config:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
rabbitmq-rabbitmq-serviceaccount-661e-token-tl4hr:
Type: Secret (a volume populated by a Secret)
SecretName: rabbitmq-rabbitmq-serviceaccount-661e-token-tl4hr
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 30m (x3 over 30m) default-scheduler 0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.
Normal Scheduled 30m default-scheduler Successfully assigned default/rabbitmq-rabbitmq-1 to gke-cluster-1-default-pool-46d75799-qzvs
Normal SuccessfulAttachVolume 30m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-a8302b5b-debc-4aa1-89bf-2d3295a201fe"
Normal Pulling 30m kubelet Pulling image "gcr.io/cloud-marketplace/google/rabbitmq/debian9:3.9.13-20220220-155313"
Normal Pulled 30m kubelet Successfully pulled image "gcr.io/cloud-marketplace/google/rabbitmq/debian9:3.9.13-20220220-155313" in 3.88917291s
Normal Created 30m kubelet Created container copy-rabbitmq-config
Normal Started 30m kubelet Started container copy-rabbitmq-config
Normal Pulling 30m kubelet Pulling image "gcr.io/cloud-marketplace/google/rabbitmq:3.9.13-20220220-155313"
Normal Pulled 29m kubelet Successfully pulled image "gcr.io/cloud-marketplace/google/rabbitmq:3.9.13-20220220-155313" in 16.874411606s
Normal Created 29m kubelet Created container rabbitmq
Normal Started 29m kubelet Started container rabbitmq
Normal Pulling 29m kubelet Pulling image "gcr.io/cloud-marketplace/google/rabbitmq/prometheus-to-sd:3.9.13-20220220-155313"
Normal Pulled 29m kubelet Successfully pulled image "gcr.io/cloud-marketplace/google/rabbitmq/prometheus-to-sd:3.9.13-20220220-155313" in 1.601774667s
Normal Created 29m kubelet Created container prometheus-to-sd
Normal Started 29m kubelet Started container prometheus-to-sd
With service as NodePort I'm able to connect to the amqp port but now the some times it connects and sometimes it doesnt. If i run my sample test it connects -> publishes the message and terminates. Then again when i run it doesnt connect. When I try after waiting a minute it connects again.
Seems like whenever the connection is terminated rabbitmq needs time to listen again.
I haven't been able to connect at all with service type loadbalancer yet.