I'm setting up a cluster with two nodes of web applications. They have a shared cache using Hazelcast 3.10.4. I deploy them on Kubernetes. The configuration for network/join is as following:
<tcp-ip enabled="true">
<member-list>
<member>10.150.28.47:5701</member>
<member>10.150.28.48:5701</member>
</member-list>
</tcp-ip>
They cannot join the cluster. The above IPs are the external IP addresses. The logs are:
[10.233.2.124]:5701 [dev] [3.10.4] Accepting socket connection from /10.150.28.6:53192
2018-12-10 16:48:59.520 INFO 7 --- [thread-Acceptor] c.h.nio.tcp.TcpIpConnectionManager : [10.233.2.124]:5701 [dev] [3.10.4] Established socket connection between /10.233.2.124:5701 and /10.150.28.6:53192
2018-12-10 16:48:59.536 WARN 7 --- [.IO.thread-in-2] c.h.nio.tcp.TcpIpConnectionManager : [10.233.2.124]:5701 [dev] [3.10.4] Wrong bind request from [10.233.2.126]:5701! This node is not the requested endpoint: [10.150.28.47]:5701
2018-12-10 16:48:59.536 INFO 7 --- [.IO.thread-in-2] com.hazelcast.nio.tcp.TcpIpConnection : [10.233.2.124]:5701 [dev] [3.10.4] Connection[id=6, /10.233.2.124:5701->/10.150.28.6:53192, endpoint=null, alive=false, type=MEMBER] closed. Reason: Wrong bind request from [10.233.2.126]:5701! This node is not the requested endpoint: [10.150.28.47]:5701
2018-12-10 16:49:00.522 INFO 7 --- [thread-Acceptor] com.hazelcast.nio.tcp.TcpIpAcceptor : [10.233.2.124]:5701 [dev] [3.10.4] Accepting socket connection from /10.150.28.6:38391
2018-12-10 16:49:00.524 INFO 7 --- [thread-Acceptor] c.h.nio.tcp.TcpIpConnectionManager : [10.233.2.124]:5701 [dev] [3.10.4] Established socket connection between /10.233.2.124:5701 and /10.150.28.6:38391
2018-12-10 16:49:00.527 WARN 7 --- [.IO.thread-in-0] c.h.nio.tcp.TcpIpConnectionManager : [10.233.2.124]:5701 [dev] [3.10.4] Wrong bind request from [10.233.2.126]:5701! This node is not the requested endpoint: [10.150.28.47]:5701
2018-12-10 16:49:00.527 INFO 7 --- [.IO.thread-in-0] com.hazelcast.nio.tcp.TcpIpConnection : [10.233.2.124]:5701 [dev] [3.10.4] Connection[id=7, /10.233.2.124:5701->/10.150.28.6:38391, endpoint=null, alive=false, type=MEMBER] closed. Reason: Wrong bind request from [10.233.2.126]:5701! This node is not the requested endpoint: [10.150.28.47]:5701
2018-12-10 16:49:01.523 INFO 7 --- [thread-Acceptor] com.hazelcast.nio.tcp.TcpIpAcceptor : [10.233.2.124]:5701 [dev] [3.10.4] Accepting socket connection from /10.150.28.6:41764
2018-12-10 16:49:01.525 INFO 7 --- [thread-Acceptor] c.h.nio.tcp.TcpIpConnectionManager : [10.233.2.124]:5701 [dev] [3.10.4] Established socket connection between /10.233.2.124:5701 and /10.150.28.6:41764
2018-12-10 16:49:01.528 WARN 7 --- [.IO.thread-in-1] c.h.nio.tcp.TcpIpConnectionManager : [10.233.2.124]:5701 [dev] [3.10.4] Wrong bind request from [10.233.2.126]:5701! This node is not the requested endpoint: [10.150.28.47]:5701
2018-12-10 16:49:01.529 INFO 7 --- [.IO.thread-in-1] com.hazelcast.nio.tcp.TcpIpConnection : [10.233.2.124]:5701 [dev] [3.10.4] Connection[id=8, /10.233.2.124:5701->/10.150.28.6:41764, endpoint=null, alive=false, type=MEMBER] closed. Reason: Wrong bind request from [10.233.2.126]:5701! This node is not the requested endpoint: [10.150.28.47]:5701
2018-12-10 16:49:02.524 INFO 7 --- [thread-Acceptor] com.hazelcast.nio.tcp.TcpIpAcceptor : [10.233.2.124]:5701 [dev] [3.10.4] Accepting socket connection from /10.150.28.6:50909
2018-12-10 16:49:02.526 INFO 7 --- [thread-Acceptor] c.h.nio.tcp.TcpIpConnectionManager : [10.233.2.124]:5701 [dev] [3.10.4] Established socket connection between /10.233.2.124:5701 and /10.150.28.6:50909
2018-12-10 16:49:02.528 WARN 7 --- [.IO.thread-in-2] c.h.nio.tcp.TcpIpConnectionManager : [10.233.2.124]:5701 [dev] [3.10.4] Wrong bind request from [10.233.2.126]:5701! This node is not the requested endpoint: [10.150.28.47]:5701
2018-12-10 16:49:02.528 INFO 7 --- [.IO.thread-in-2] com.hazelcast.nio.tcp.TcpIpConnection : [10.233.2.124]:5701 [dev] [3.10.4] Connection[id=9, /10.233.2.124:5701->/10.150.28.6:50909, endpoint=null, alive=false, type=MEMBER] closed. Reason: Wrong bind request from [10.233.2.126]:5701! This node is not the requested endpoint: [10.150.28.47]:5701
2018-12-10 16:49:03.524 INFO 7 --- [thread-Acceptor] com.hazelcast.nio.tcp.TcpIpAcceptor : [10.233.2.124]:5701 [dev] [3.10.4] Accepting socket connection from /10.150.28.6:59191
2018-12-10 16:49:03.526 INFO 7 --- [thread-Acceptor] c.h.nio.tcp.TcpIpConnectionManager : [10.233.2.124]:5701 [dev] [3.10.4] Established socket connection between /10.233.2.124:5701 and /10.150.28.6:59191
If you are on K8s environment, have you tried using Hazelcast Kubernetes discovery plugin?