For sharing events between the pods of two different services in a Kubernetes namespace, I intend to use Hazelcast. This is not a problem, however, each service also has a cluster that contains all its pods.

So, I have two clusters using the same pods. I achieved separation of the clusters by setting a group name for one of the clusters, while the other has the default group configuration. This works fine locally, with multiple instances of a test application. However, this is with multicast enabled.

In Kubernetes however, Hazelcast uses the HazelcastKubernetesDiscoveryStrategy and has Multicast disabled.

Both services have a label:

metadata:
  name: service-1
  labels:
    hazelcast-group: bc-events

metadata:
  name: service-2
  labels:
    hazelcast-group: bc-events

and the hazelcast configuration for the events cluster is like this:

Config hzConfig = new Config("events-instance");

NetworkConfig nwConfig = new NetworkConfig();
JoinConfig joinConfig = new JoinConfig();
joinConfig.setMulticastConfig(new MulticastConfig().setEnabled(false));
joinConfig.setTcpIpConfig(new TcpIpConfig().setEnabled(false));

DiscoveryStrategyConfig k8sDiscoveryStrategy = new DiscoveryStrategyConfig("com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy");
k8sDiscoveryStrategy.addProperty("namespace", "dev");
k8sDiscoveryStrategy.addProperty("resolve-not-ready-addresses", true);
k8sDiscoveryStrategy.addProperty("service-label-name", "hazelcast-group");
k8sDiscoveryStrategy.addProperty("service-label-value", "bc-events");

DiscoveryConfig discoveryConfig = new DiscoveryConfig();
discoveryConfig.addDiscoveryStrategyConfig(k8sDiscoveryStrategy);
joinConfig.setDiscoveryConfig(discoveryConfig);
nwConfig.setJoin(joinConfig);

hzConfig.setNetworkConfig(nwConfig);
hzConfig.setProperty("hazelcast.discovery.enabled", "true");

GroupConfig groupConfig = new GroupConfig("bc-events");
hzConfig.setGroupConfig(groupConfig);

while the configuration for the shared cache cluster (the one without a group) is like this (for service 1, service 2 is the same):

Config hzConfig = new Config("service-1-app-hc");

NetworkConfig nwConfig = new NetworkConfig();
JoinConfig joinConfig = new JoinConfig();
joinConfig.setMulticastConfig(new MulticastConfig().setEnabled(false));
joinConfig.setTcpIpConfig(new TcpIpConfig().setEnabled(false));

DiscoveryStrategyConfig k8sDiscoveryStrategy = new DiscoveryStrategyConfig("com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy");

k8sDiscoveryStrategy.addProperty("namespace", "dev");
k8sDiscoveryStrategy.addProperty("service-name", "service-1");
k8sDiscoveryStrategy.addProperty("resolve-not-ready-addresses", true);

DiscoveryConfig discoveryConfig = new DiscoveryConfig();
discoveryConfig.addDiscoveryStrategyConfig(k8sDiscoveryStrategy);
joinConfig.setDiscoveryConfig(discoveryConfig);
nwConfig.setJoin(joinConfig);

hzConfig.setNetworkConfig(nwConfig);
hzConfig.setProperty("hazelcast.discovery.enabled", "true");

The hazelcast instances find eachother, but then complain that the other has a different group name and blacklists the IP.

While debugging the code, during config validation while processing a join request, it tries to compare the group names (bc-events against dev) and obviously that's different. But then this gets blacklisted (I believe), preventing a validation check for the other instance that has the same group name.

I'm not sure where to go next. I cannot test this configuration locally because without multicast, it doesn't find the other nodes for joining the cluster. I also don't think there is anything wrong with the configuration.

The libraries used are:

<dependency>
    <groupId>com.hazelcast</groupId>
    <artifactId>hazelcast</artifactId>
    <version>3.7.8</version>
</dependency>
<dependency>
    <groupId>com.hazelcast</groupId>
    <artifactId>hazelcast-kubernetes</artifactId>
    <version>1.1.0</version>
</dependency>

UPDATE:

I should note that, the current setup, where this cluster that has the group name discovers by service name (as such, it only contains the pods of one service) actually works. The cluster without a group and the cluster with a group are running alongside eachother. It is only when I switch to label-based discovery (and that the other service gets involved) that it breaks.

UPDATE:

When changing the port of the events cluster, I noticed it still tries to connect to 5701 (the default) despite being put on 5801. Of course this works because the first cluster is running on 5701. Inside HazelcastKubernetesDiscoveryStrategy there is the following method:

protected int getServicePort(Map<String, Object> properties) {
    int port = NetworkConfig.DEFAULT_PORT;
    if (properties != null) {
        String servicePort = (String) properties.get(HAZELCAST_SERVICE_PORT);
        if (servicePort != null) {
            port = Integer.parseInt(servicePort);
        }
    }
    return port;
}

This method checks the additional properties that get returned for each endpoint returned by the kubernetes client, for a hazelcast port configuration. If none exists, it uses the default 5701. I am guessing this is what needs to be configured, however, it must not impact the other cluster, so I may have to extend the strategy with some of my own logic.