I'm following this instruction to setup glusterfs on my kubernetes cluster. At heketi-client/bin/heketi-cli setup-openshift-heketi-storage part, heketi-cli tells me :

Error: volume create: heketidbstorage: failed: Host 192.168.99.25 not connected

or sometimes:

Error: volume create: heketidbstorage: failed: Staging failed on 192.168.99.26. Error: Host 192.168.99.25 not connected

heketi.json is

{
  "_port_comment": "Heketi Server Port Number",
  "port": "8080",

  "_use_auth": "Enable JWT authorization. Please enable for deployment",
  "use_auth": false,

  "_jwt": "Private keys for access",
  "jwt": {
    "_admin": "Admin has access to all APIs",
    "admin": {
      "key": "7319"
    },
    "_user": "User only has access to /volumes endpoint",
    "user": {
      "key": "7319"
    }
  },

  "_glusterfs_comment": "GlusterFS Configuration",
  "glusterfs": {
    "_executor_comment": "Execute plugin. Possible choices: mock, kubernetes, ssh",
    "executor": "kubernetes",

    "_db_comment": "Database file name",
    "db": "/var/lib/heketi/heketi.db",

    "kubeexec": {
      "rebalance_on_expansion": true
    },

    "sshexec": {
      "rebalance_on_expansion": true,
      "keyfile": "/etc/heketi/private_key",
      "fstab": "/etc/fstab",
      "port": "22",
      "user": "root",
      "sudo": false
    }
  },

  "_backup_db_to_kube_secret": "Backup the heketi database to a Kubernetes secret when running in Kubernetes. Default is off.",
  "backup_db_to_kube_secret": false
}

topology-sample.json is

{
  "clusters": [
    {
      "nodes": [
        {
          "node": {
            "hostnames": {
              "manage": [
                "redis-test25"
              ],
              "storage": [
                "192.168.99.25"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sda7",
              "destroydata": true
            }
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "redis-test26"
              ],
              "storage": [
                "192.168.99.26"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sda7",
              "destroydata": true
            }
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "redis-test01"
              ],
              "storage": [
                "192.168.99.113"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sda7",
              "destroydata": true
            }
          ]
        }
      ]
    }
  ]
}

The heketi-cli is v8.0.0 and kubernetes is v1.12.3

How do I fix this problem?

Update: Just found that I missed the iptables part, but now the message becomes

Error: volume create: heketidbstorage: failed: Host 192.168.99.25 is not in 'Peer in Cluster' state

seems that one of the glusterfs pod cannot connect to others, I tried kubectl exec -i glusterfs-59ftx -- gluster peer status:

Number of Peers: 2

Hostname: 192.168.99.26
Uuid: 6950db9a-3d60-4625-b642-da5882396bee
State: Peer Rejected (Disconnected)

Hostname: 192.168.99.113
Uuid: 78983466-4499-48d2-8411-2c3e8c70f89f
State: Peer Rejected (Disconnected)

while the other one said:

Number of Peers: 1

Hostname: 192.168.99.26
Uuid: 23a0114d-65b8-42d6-8067-7efa014af68d
State: Peer in Cluster (Connected)