Rabbitmq peer discovery on k8s failed if k8s API tls cipher defined

8/21/2018

I use a helm chart to deploy 2 RabbitMQ pods on k8s. The chart can be deployed well and cluster well firstly. Then add a tls definition for k8s API:

--tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

Then the RabbitMQ peer discovery plugin failed to cluster.

[root@control-01]$ # kubectl get pod -o wide
oe-crmq-0                                   0/1       CrashLoopBackOff   7          33m       192.168.1.186   worker-01
oe-crmq-1                                   0/1       CrashLoopBackOff   7          32m       192.168.1.105   worker-02
[root@control-01]$ # kubectl logs oe-crmq-0
  ##  ##
  ##  ##      RabbitMQ 3.7.5. Copyright (C) 2007-2018 Pivotal Software, Inc.
  ##########  Licensed under the MPL.  See http://www.rabbitmq.com/
  ######  ##
  ##########  Logs: /var/log/rabbitmq/rabbit@oe-crmq-0.log
                    /var/log/rabbitmq/rabbit@oe-crmq-0_upgrade.log

              Starting broker...
{"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{case_clause,{error,\"{failed_connect,[{to_address,{\\"kubernetes.default.svc.cluster.local\\",8443}},\n                 {inet,[inet],etimedout}]}\"}},[{rabbit_mnesia,init_from_config,0,[{file,\"src/rabbit_mnesia.erl\"},{line,164}]},{rabbit_mnesia,init_with_lock,3,[{file,\"src/rabbit_mnesia.erl\"},{line,144}]},{rabbit_mnesia,init,0,[{file,\"src/rabbit_mnesia.erl\"},{line,111}]},{rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,[{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},{rabbit_boot_steps,run_step,2,[{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},{rabbit_boot_steps,'-run_boot_steps/1-lc$^0/1-0-',1,[{file,\"src/rabbit_boot_steps.erl\"},{line,26}]},{rabbit_boot_steps,run_boot_steps,1,[{file,\"src/rabbit_boot_steps.erl\"},{line,26}]},{rabbit,start,2,[{file,\"src/rabbit.erl\"},{line,801}]}]}}}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{case_clause,{error,"{failed_connect,[{to_address,{\"kubernetes.defau

Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done

So I try to add cipher for RabbitMQ in the advanced.config:

bash-4.2$ cat advanced.config
%% List allowed ciphers
[
    {ssl, [{versions, ['tlsv1.2']},
           {ssl_optons, [{ciphers, [
                            {ecdhe_rsa,aes_128_gcm,null,sha256}
                            ]}, {fail_if_no_peer_cert,false}]}]}
].

While this config haven't bring much improvemnt. It still timeout when connect to kubernetes.default.svc.cluster.local.

-- Haiyan GU
kubernetes
rabbitmq
ssl

1 Answer

4/24/2019

After upgrading erlang to latest version the issue resolved.

-- Haiyan GU
Source: StackOverflow