Curious behavior with TLS RabbitMQ in Kubernetes and externalIP

1/29/2020

I deployed a RabbitMQ (from the helm chart https://github.com/helm/charts/tree/master/stable/rabbitmq) in a Kubernetes cluster, in namespace named rabbitmq. I added 3 IPs (the IPs of my Kubernetes nodes) as externalIPs in rabbitmq service. Here is the rabbitmq service :

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: rabbitmq
    app.kubernetes.io/name: rabbitmq
    helm.sh/chart: rabbitmq-6.15.0
  name: rabbitmq
  namespace: rabbitmq
spec:
  externalIPs:
  - x.x.x.x1
  - x.x.x.x2
  - x.x.x.x3
  ports:
  - name: epmd
    port: 4369
    protocol: TCP
    targetPort: epmd
  - name: amqp
    port: 5672
    protocol: TCP
    targetPort: amqp
  - name: amqp-ssl
    port: 5671
    protocol: TCP
    targetPort: amqp-ssl
  - name: dist
    port: 25672
    protocol: TCP
    targetPort: dist
  - name: stats
    port: 15672
    protocol: TCP
    targetPort: stats
  - name: metrics
    port: 9419
    protocol: TCP
    targetPort: metrics
  selector:
    app.kubernetes.io/component: rabbitmq
    app.kubernetes.io/name: rabbitmq
    helm.sh/chart: rabbitmq-6.15.0
  type: ClusterIP

I have an external load balancer which target these 3 IPs. I have a DNS entry "rabbitmq.mycompagny.com" which target my load balancer. I always use this mecanism to target all my ingresses without any problem.

Finally, I have a test pod in a namespace test, with amqp-tools installed. On this test pod, see some commands and results :

amqp-get --url=amqps://user:password@rabbitmq.rabbitmq.svc.cluster.local:5671 --cacert=/opt/bitnami/rabbitmq/certs/ca_certificate.pem --key=/opt/bitnami/rabbitmq/certs/server_key.pem --cert=/opt/bitnami/rabbitmq/certs/server_certificate.pem --queue=test --ssl
# opening socket to rabbitmq.rabbitmq.svc.cluster.local:5671

# => KO

# logs from rabbitmq :
#2020-01-29 07:34:25.308 [debug] <0.3077.0> accepting AMQP connection <0.3077.0> (10.244.96.29:42620 -> 10.244.32.52:5671)
#2020-01-29 07:34:25.308 [debug] <0.3077.0> closing AMQP connection <0.3077.0> (10.244.96.29:42620 -> 10.244.32.52:5671):
#connection_closed_with_no_data_received

amqp-get --url=amqps://user:password@rabbitmq.mycompagny.com:5671 --cacert=/opt/bitnami/rabbitmq/certs/ca_certificate.pem --key=/opt/bitnami/rabbitmq/certs/server_key.pem --cert=/opt/bitnami/rabbitmq/certs/server_certificate.pem --queue=test --ssl

# => OK

# logs from rabbitmq :
#2020-01-29 07:37:47.430 [info] <0.23936.4> accepting AMQP connection <0.23936.4> (10.244.32.0:55694 -> 10.244.32.28:5671)
#2020-01-29 07:37:47.472 [debug] <0.23936.4> User 'user' authenticated successfully by backend rabbit_auth_backend_internal
#2020-01-29 07:37:47.486 [info] <0.23936.4> closing AMQP connection <0.23936.4> (10.244.32.0:55694 -> 10.244.32.28:5671, vhost: '/', user: 'user')

amqp-get --url=amqp://user:password@rabbitmq.rabbitmq.svc.cluster.local:5672 --queue=test

# => OK

amqp-get --url=amqp://user:password@rabbitmq.mycompagny.com:5672 --queue=test
#opening socket to rabbitmq.mycompagny.fr:5672

# => OK

The difference between these commands is the target DNS. With TLS, when I use the external DNS, it's ok, but it's not the case with the internal DNS. Can you explain why ?

I double checked the certificate, which is a signed wildcard certificate (check with openssl s_server/s_client).

I have to use externalIPs focused by an external load balancer to access rabbitmq since Kubernetes ingresses only support HTTP/HTTPS protocol and the rabbitmq protocol is AMQP/AMPQS.

-- Antoine
kubernetes
rabbitmq
ssl

0 Answers