I am trying to make an Hazel cast connection using the Hazel Cast Java client from the spark job running in the spark cluster. The Hazelcast is running as a pod with in the kubernetes cluster and i have exposed the Hazelcast service to be accessible from outside the kubernetes cluster using node port.

Facing the problem when there are more than one Hazel cast pods being running in different Kubernetes cluster nodes which internally gets some private IP.

Below is the Java Hazel cast client configuration that i am using:

cc = new ClientConfig()

var networkConfig:ClientNetworkConfig = cc.getNetworkConfig()

networkConfig.addAddress(kube_node1_IP,kube_node2_IP,kube_node3_IP)

networkConfig.setConnectionAttemptLimit(500)

cc.getSerializationConfig().addSerializerConfig(new SerializerConfig()
      .setImplementation(new EventSerializer()).setTypeClass(EventClass))
      .addPortableFactoryClass(1, keyFactoryClass)

The exception back trace is as follows:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0

com.hazelcast.core.HazelcastException: java.io.IOException: No available connection to address Address[10.2.36.5]:5701 at com.hazelcast.util.ExceptionUtil.peel(ExceptionUtil.java:73) at com.hazelcast.util.ExceptionUtil.peel(ExceptionUtil.java:63) at com.hazelcast.util.ExceptionUtil.peel(ExceptionUtil.java:52) at com.hazelcast.util.ExceptionUtil.rethrow(ExceptionUtil.java:83) at com.hazelcast.client.spi.ClientProxy.invokeOnPartition(ClientProxy.java:128) at com.hazelcast.client.spi.ClientProxy.invoke(ClientProxy.java:120) at com.hazelcast.client.proxy.ClientMapProxy.getInternal(ClientMapProxy.java:225) at com.hazelcast.client.proxy.ClientMapProxy.get(ClientMapProxy.java:220)

Here 10.2.36.5 is the private IP of one of the hazel cast pod.

P.S. : In case there is only single pod, there is no issue with the same configuration.