Embeded Hazelcast cannot join on Kubernetes cluster

12/10/2018

I'm setting up a cluster with two nodes of web applications. They have a shared cache using Hazelcast 3.10.4. I deploy them on Kubernetes. The configuration for network/join is as following:

<tcp-ip enabled="true">
       <member-list>
            <member>10.150.28.47:5701</member>
            <member>10.150.28.48:5701</member>
       </member-list>
</tcp-ip>

They cannot join the cluster. The above IPs are the external IP addresses. The logs are:

[10.233.2.124]:5701 [dev] [3.10.4] Accepting socket connection from /10.150.28.6:53192
2018-12-10 16:48:59.520  INFO 7 --- [thread-Acceptor] c.h.nio.tcp.TcpIpConnectionManager       : [10.233.2.124]:5701 [dev] [3.10.4] Established socket connection between /10.233.2.124:5701 and /10.150.28.6:53192
2018-12-10 16:48:59.536  WARN 7 --- [.IO.thread-in-2] c.h.nio.tcp.TcpIpConnectionManager       : [10.233.2.124]:5701 [dev] [3.10.4] Wrong bind request from [10.233.2.126]:5701! This node is not the requested endpoint: [10.150.28.47]:5701
2018-12-10 16:48:59.536  INFO 7 --- [.IO.thread-in-2] com.hazelcast.nio.tcp.TcpIpConnection    : [10.233.2.124]:5701 [dev] [3.10.4] Connection[id=6, /10.233.2.124:5701->/10.150.28.6:53192, endpoint=null, alive=false, type=MEMBER] closed. Reason: Wrong bind request from [10.233.2.126]:5701! This node is not the requested endpoint: [10.150.28.47]:5701
2018-12-10 16:49:00.522  INFO 7 --- [thread-Acceptor] com.hazelcast.nio.tcp.TcpIpAcceptor      : [10.233.2.124]:5701 [dev] [3.10.4] Accepting socket connection from /10.150.28.6:38391
2018-12-10 16:49:00.524  INFO 7 --- [thread-Acceptor] c.h.nio.tcp.TcpIpConnectionManager       : [10.233.2.124]:5701 [dev] [3.10.4] Established socket connection between /10.233.2.124:5701 and /10.150.28.6:38391
2018-12-10 16:49:00.527  WARN 7 --- [.IO.thread-in-0] c.h.nio.tcp.TcpIpConnectionManager       : [10.233.2.124]:5701 [dev] [3.10.4] Wrong bind request from [10.233.2.126]:5701! This node is not the requested endpoint: [10.150.28.47]:5701
2018-12-10 16:49:00.527  INFO 7 --- [.IO.thread-in-0] com.hazelcast.nio.tcp.TcpIpConnection    : [10.233.2.124]:5701 [dev] [3.10.4] Connection[id=7, /10.233.2.124:5701->/10.150.28.6:38391, endpoint=null, alive=false, type=MEMBER] closed. Reason: Wrong bind request from [10.233.2.126]:5701! This node is not the requested endpoint: [10.150.28.47]:5701
2018-12-10 16:49:01.523  INFO 7 --- [thread-Acceptor] com.hazelcast.nio.tcp.TcpIpAcceptor      : [10.233.2.124]:5701 [dev] [3.10.4] Accepting socket connection from /10.150.28.6:41764
2018-12-10 16:49:01.525  INFO 7 --- [thread-Acceptor] c.h.nio.tcp.TcpIpConnectionManager       : [10.233.2.124]:5701 [dev] [3.10.4] Established socket connection between /10.233.2.124:5701 and /10.150.28.6:41764
2018-12-10 16:49:01.528  WARN 7 --- [.IO.thread-in-1] c.h.nio.tcp.TcpIpConnectionManager       : [10.233.2.124]:5701 [dev] [3.10.4] Wrong bind request from [10.233.2.126]:5701! This node is not the requested endpoint: [10.150.28.47]:5701
2018-12-10 16:49:01.529  INFO 7 --- [.IO.thread-in-1] com.hazelcast.nio.tcp.TcpIpConnection    : [10.233.2.124]:5701 [dev] [3.10.4] Connection[id=8, /10.233.2.124:5701->/10.150.28.6:41764, endpoint=null, alive=false, type=MEMBER] closed. Reason: Wrong bind request from [10.233.2.126]:5701! This node is not the requested endpoint: [10.150.28.47]:5701
2018-12-10 16:49:02.524  INFO 7 --- [thread-Acceptor] com.hazelcast.nio.tcp.TcpIpAcceptor      : [10.233.2.124]:5701 [dev] [3.10.4] Accepting socket connection from /10.150.28.6:50909
2018-12-10 16:49:02.526  INFO 7 --- [thread-Acceptor] c.h.nio.tcp.TcpIpConnectionManager       : [10.233.2.124]:5701 [dev] [3.10.4] Established socket connection between /10.233.2.124:5701 and /10.150.28.6:50909
2018-12-10 16:49:02.528  WARN 7 --- [.IO.thread-in-2] c.h.nio.tcp.TcpIpConnectionManager       : [10.233.2.124]:5701 [dev] [3.10.4] Wrong bind request from [10.233.2.126]:5701! This node is not the requested endpoint: [10.150.28.47]:5701
2018-12-10 16:49:02.528  INFO 7 --- [.IO.thread-in-2] com.hazelcast.nio.tcp.TcpIpConnection    : [10.233.2.124]:5701 [dev] [3.10.4] Connection[id=9, /10.233.2.124:5701->/10.150.28.6:50909, endpoint=null, alive=false, type=MEMBER] closed. Reason: Wrong bind request from [10.233.2.126]:5701! This node is not the requested endpoint: [10.150.28.47]:5701
2018-12-10 16:49:03.524  INFO 7 --- [thread-Acceptor] com.hazelcast.nio.tcp.TcpIpAcceptor      : [10.233.2.124]:5701 [dev] [3.10.4] Accepting socket connection from /10.150.28.6:59191
2018-12-10 16:49:03.526  INFO 7 --- [thread-Acceptor] c.h.nio.tcp.TcpIpConnectionManager       : [10.233.2.124]:5701 [dev] [3.10.4] Established socket connection between /10.233.2.124:5701 and /10.150.28.6:59191 
-- Minh Trần
hazelcast
kubernetes

1 Answer

12/10/2018

If you are on K8s environment, have you tried using Hazelcast Kubernetes discovery plugin?

-- sertug
Source: StackOverflow