I have 5 replicas of stateful set of Apache Ignite cluster into a Kubernetes. I use TcpDiscoveryKubernetesIpFinder as a node discovery mechanism in the cluster.

After running the cluster for sometime, I found out that the cluster is being split into 2 separated topologies.

Below is an example of config.xml of Apache Ignite:

<?xml version="1.0" encoding="UTF-8"?>

<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->

<!--
    Configuration example with Kubernetes IP finder and Ignite persistence enabled.
    WAL files and database files are stored in separate disk drives.
-->
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="
        http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd">

    <bean class="org.apache.ignite.configuration.IgniteConfiguration">

        <property name="dataStorageConfiguration">
            <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
                <!-- Redefining the default region's settings -->
                <property name="defaultDataRegionConfiguration">
                <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
                    <property name="name" value="default"/>
                    <!-- Setting the size of the default region to 4GB. -->
                    <property name="maxSize" value="#{256L * 1024 * 1024}"/>
                    <!-- Enabling RANDOM_LRU eviction for this region.  -->
                    <property name="pageEvictionMode" value="RANDOM_LRU"/>
                </bean>
                </property>
            </bean>
        </property>

        <property name="peerClassLoadingEnabled" value="true"/>
        <!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
        <property name="discoverySpi">
            <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ipFinder">
                    <!--
                        Enables Kubernetes IP finder and setting custom namespace and service names.
                    -->
                    <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder">
                        <property name="namespace" value="dev"/>
                        <property name="serviceName" value="ignite-dev"/>
                    </bean>
                </property>
            </bean>
        </property>
    </bean>
</beans>

I would like to know how can I prevent this from happening?