I have tried to make a redis cluster in k8s environment using "NodePort" type of service. More specifically, I want to compose a redis cluster across two different k8s cluster.

When I used LoadBalancer(External IP) for service type, cluster was made successfully. The problem is NodePort.

After I command redis-cli --cluster create, it stucks on "Waiting for the cluster to join"

Below is the logs of cluster create command. I deployed 4 leader pods and 4 slave pods with individual nodeport service.

root@redis-leader00-5fc546c4bd-28x8w:/data# redis-cli -a mypassword --cluster create --cluster-replicas 1 \
> 192.168.9.194:30030 192.168.9.199:30031 192.168.9.194:30032 192.168.9.199:30033 \
> 192.168.9.199:30030 192.168.9.194:30031 192.168.9.199:30032 192.168.9.194:30033
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 8 nodes...
Master[0] -> Slots 0 - 4095
Master[1] -> Slots 4096 - 8191
Master[2] -> Slots 8192 - 12287
Master[3] -> Slots 12288 - 16383
Adding replica 192.168.9.199:30030 to 192.168.9.194:30030
Adding replica 192.168.9.194:30033 to 192.168.9.199:30031
Adding replica 192.168.9.199:30032 to 192.168.9.194:30032
Adding replica 192.168.9.194:30031 to 192.168.9.199:30033
M: 94bf3c6760e6b3a91c408eda497822b4961e8d82 192.168.9.194:30030
   slots:[0-4095] (4096 slots) master
M: 31f4a9604b15109316f91956aa4a32b0c6952a4d 192.168.9.199:30031
   slots:[4096-8191] (4096 slots) master
M: 0738d1e1a677352fc3b0b3600a67d837b795fa8a 192.168.9.194:30032
   slots:[8192-12287] (4096 slots) master
M: 7dd7edbfab6952273460778d1f140b0716118042 192.168.9.199:30033
   slots:[12288-16383] (4096 slots) master
S: 17e044681319d0a05bd92deeb4ead31c0cd468e2 192.168.9.199:30030
   replicates 94bf3c6760e6b3a91c408eda497822b4961e8d82
S: 9c9e47ec566ac781e8e3dcb51398a27d1da71004 192.168.9.194:30031
   replicates 7dd7edbfab6952273460778d1f140b0716118042
S: b8f7028b56f96565a91fdb442c94fbedcee088c2 192.168.9.199:30032
   replicates 0738d1e1a677352fc3b0b3600a67d837b795fa8a
S: e4c9ffdf67e2b2ef9f840618110738358bde52d5 192.168.9.194:30033
   replicates 31f4a9604b15109316f91956aa4a32b0c6952a4d
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
...........................................

The weird point is that the other redis containers received the signal but there is no progress. Below is the logs of other redis containers.

.... // Some logs for initializing redis
1:M 20 Jan 2022 06:09:12.055 * Ready to accept connections
1:M 20 Jan 2022 06:13:41.263 # configEpoch set to 5 via CLUSTER SET-CONFIG-EPOCH

I thought the communication was successful but gossip port didn't work properly. So, I modified redis.conf and set cluster-announce-bus-port but it also didn't work.

How can I compose a redis cluster using nodeport type of service?

Please refer the one of .yaml files

apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-cluster-leader00
  namespace: redis
  labels:
    app: redis-cluster
    leader: "00"
data:
  fix-ip.sh: |
    #!/bin/sh
    CLUSTER_CONFIG="/data/nodes.conf"
    if [ -f ${CLUSTER_CONFIG} ]; then
      if [ -z "${HOST_IP}" ]; then
        echo "Unable to determine Pod IP address!"
        exit 1
      fi
      echo "Updating my IP to ${HOST_IP} in ${CLUSTER_CONFIG}"
      sed -i.bak -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${HOST_IP}/" ${CLUSTER_CONFIG}
    fi
    exec "$@"
  redis.conf: |+
    bind 0.0.0.0
    cluster-enabled yes
    cluster-require-full-coverage no
    cluster-node-timeout 15000
    cluster-config-file /data/nodes.conf
    cluster-migration-barrier 1
    appendonly no
    save ""
    protected-mode no
    requirepass "mypassword"
    masterauth "mypassword"
    cluster-announce-ip 192.168.9.194
    cluster-announce-port 30030
    cluster-announce-bus-port 31030
---
apiVersion: v1
kind: Service
metadata:
  name: redis-leader00
  namespace: redis
  labels:
    app: redis
    role: leader
    tier: backend
    leader: "00"
spec:
  ports:
  - port: 6379
    targetPort: 6379
    nodePort: 30030
    name: client
  - port: 16379
    targetPort: 16379
    nodePort: 31030
    name: gossip
  selector:
    app: redis
    role: leader
    tier: backend
    leader: "00"
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-leader00
  namespace: redis
  labels:
    app: redis
    role: leader
    tier: backend
    leader: "00"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
      leader: "00"
  template:
    metadata:
      labels:
        app: redis
        role: leader
        tier: backend
        leader: "00"
    spec:
      containers:
      - name: leader
        image: docker.io/redis:6.0.5
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 6379
          name: client
        - containerPort: 16379
          name: gossip
        volumeMounts:
        - name: conf
          mountPath: /conf
          readOnly: false
        args: ["--requirepass", "mypassword"]
        command: ["/conf/fix-ip.sh", "redis-server", "/conf/redis.conf"]
        env:
        - name: HOST_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
      volumes:
      - name: conf
        configMap:
          name: redis-cluster-leader00
          defaultMode: 0755

Also see nodes.conf file in container after I commanded redis-cli create

root@redis-leader01-87ccb466-bsnq4:/data# cat nodes.conf
31f4a9604b15109316f91956aa4a32b0c6952a4d 192.168.9.199:30031@31031 myself,master - 0 0 2 connected 4096-8191
vars currentEpoch 2 lastVoteEpoch 0