I am trying to run my grails application with ehcache in kubernetes. When I ran the docker image in my local system with 2 containers, it is able to replicate the cache between the 2 containers. Below is the log statement of ehcache heartbeat receiver thread from container 1:

 2019-12-10 16:14:49,776 [Multicast Heartbeat Receiver Thread] DEBUG net.sf.ehcache.distribution.MulticastKeepaliveHeartbeatReceiver  - rmiUrls received //172.17.0.2:40001/categories|//172.17.0.2:40001/items

Below is the log statement of ehcache heartbeat receiver thread from container 2:

2019-12-10 16:32:22,200 [Multicast Heartbeat Receiver Thread] DEBUG net.sf.ehcache.distribution.MulticastKeepaliveHeartbeatReceiver  - rmiUrls received //172.17.0.3:40001/categories|//172.17.0.3:40001/items

Deployed the same application in Kubernetes cluster with 2 pods. But application is not able to replicate the cache between 2 pods.

I was not able to find the above log statements in any of the pods. It is unable to find the peers. Below are the logs from kubernetes pods:

2019-12-10 04:25:24,136 [localhost-startStop-1] DEBUG net.sf.ehcache.Cache  - Initialised cache: categories                                                                                            
2019-12-10 04:25:24,136 [localhost-startStop-1] DEBUG net.sf.ehcache.distribution.RMIBootstrapCacheLoader  - Attempting to acquire cache peers for cache categories to bootstrap from. Will wait up to 10100ms for cache to join cluster.
2019-12-10 04:25:36,048 [Multicast Heartbeat Sender Thread] DEBUG net.sf.ehcache.distribution.PayloadUtil  - Cache peers for this CacheManager to be advertised: //192.168.108.40:40001/categories|//192.168.108.40:40001/items
2019-12-10 04:25:46,147 [localhost-startStop-1] DEBUG net.sf.ehcache.distribution.RMIBootstrapCacheLoader  - cache peers: []                                                                                
2019-12-10 04:25:46,147 [localhost-startStop-1] DEBUG net.sf.ehcache.distribution.RMIBootstrapCacheLoader  - Empty list of cache peers for cache categories. No cache peer to bootstrap from.

ehcache.xml

<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="ehcache.xsd" defaultTransactionTimeoutInSeconds="15" dynamicConfig="true" maxBytesLocalDisk="0" maxBytesLocalH

     <diskStore path="/etc/ehcache/storage" />                                                                                                                                             

     <defaultCache                                                                                                                                                                                       
            maxElementsInMemory="10000"                                                                                                                                                                  
            eternal="true"                                                                                                                                                                               
            timeToIdleSeconds="120"                                                                                                                                                                      
            timeToLiveSeconds="120"                                                                                                                                                                      
            overflowToDisk="true"                                                                                                                                                                        
            maxElementsOnDisk="10000000"                                                                                                                                                                 
            diskPersistent="false"                                                                                                                                                                       
            diskExpiryThreadIntervalSeconds="120"                                                                                                                                                        
            memoryStoreEvictionPolicy="LRU"                                                                                                                                                              
     />                                                                                                                                                                                                  

     <cacheManagerPeerProviderFactory class='net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory'                                                                                             
             properties="timeToLive=32,multicastGroupAddress=230.0.0.1,multicastGroupPort=4446,peerDiscovery=automatic"                                                                                  
             propertySeparator=','                                                                                                                                                                       
     />                                                                                                                                                                                                  

     <cacheManagerPeerListenerFactory class='net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory'                                                                                             
             properties="port=40001,socketTimeoutMillis=10000"                                                                                                                                           
             propertySeparator=','                                                                                                                                                                       
     />                                                                                                                                                                                                  

     <cache name="categories"                                                                                                                                                                          
            eternal="true"                                                                                                                                                                               
            overflowToDisk="false"                                                                                                                                                                       
            maxElementsInMemory="1000"                                                                                                                                                                   
            maxElementsOnDisk="10000000"                                                                                                                                                                 
            memoryStoreEvictionPolicy="LRU">                                                                                                                                                             

             <cacheEventListenerFactory class='net.sf.ehcache.distribution.RMICacheReplicatorFactory'                                                                                                    
                     properties="replicateAsynchronously=true"                                                                                                                                           
                     propertySeparator=','                                                                                                                                                               
             />                                                                                                                                                                                          

             <bootstrapCacheLoaderFactory class='net.sf.ehcache.distribution.RMIBootstrapCacheLoaderFactory'                                                                                             
                     properties="bootstrapAsynchronously=false"                                                                                                                                          
                     propertySeparator=','                                                                                                                                                               
             />                                                                                                                                                                                          
     </cache>


     <cache name="items"                                                                                                                                                                          
            eternal="true"                                                                                                                                                                               
            overflowToDisk="false"                                                                                                                                                                       
            maxElementsInMemory="1000"                                                                                                                                                                   
            maxElementsOnDisk="10000000"                                                                                                                                                                 
            memoryStoreEvictionPolicy="LRU">                                                                                                                                                             

             <cacheEventListenerFactory class='net.sf.ehcache.distribution.RMICacheReplicatorFactory'                                                                                                    
                     properties="replicateAsynchronously=true"                                                                                                                                           
                     propertySeparator=','                                                                                                                                                               
             />                                                                                                                                                                                          

             <bootstrapCacheLoaderFactory class='net.sf.ehcache.distribution.RMIBootstrapCacheLoaderFactory'                                                                                             
                     properties="bootstrapAsynchronously=false"                                                                                                                                          
                     propertySeparator=','                                                                                                                                                               
             />                                                                                                                                                                                          
     </cache>
</ehcache>

Kubernetes Deployment config:

apiVersion: v1
kind: Service
metadata:
  name: "mykart-service"
spec:
  ports:
    - port: 8080
      targetPort: mykart
      nodePort: 30074
      name: mykart
  selector:
    app: hushly
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mykart-deployment
spec:
  selector:
    matchLabels:
      app: mykart
  replicas: 2
  template:
    metadata:
      labels:
        app: mykart
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: nodegroupType
                    operator: In
                    values:
                      - micro-service
      volumes:
        - name: config-volume
          secret:
            secretName: mykart-config
      containers:
        - name: mykart
          image: mykart.dkr.ecr.us-east-1.amazonaws.com/mykart:12
          volumeMounts:
            - name: config-volume
              readOnly: true
              mountPath: "/etc/mykart-config"
          ports:
            - name: mykart
              containerPort: 8080