Nodes are not discovered in OrientDB cluster with Kubernetes

10/25/2018

I used this guide to setup OrientDB cluster in Kubernetes. However, it seems that each node on each pod creates its own cluster instead of joining a shared one. So the logs on each pod display such a message:

Members [1] {
    Member [pod-ip]:5701 - generated id
}

What could cause such a problem?

My orientdb-server-config file looks this way:

<handler class="com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin">
     <parameters>
          <parameter value="true" name="enabled"/>
          <parameter value="orientdb/config/default-distributed-db-config.json" name="configuration.db.default"/>
          <parameter value="orientdb/config/hazelcast.xml" name="configuration.hazelcast"/>
          <parameter name="nodeName" value="$pod_dns" />
     </parameters>
</handler>

My hazelcast.xml file looks like this (pod_dns is the name of the pod stored in env):

 </network>
    <properties>
        <property name="hazelcast.discovery.enabled">true</property>
    </properties>
    <network>
        <join>
            <multicast enabled="false"/>
            <tcp-ip enabled="false"/>
            <discovery-strategies>
                <discovery-strategy enabled="true"
                                    class="com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy">
                    <properties>
                        <property name="service-dns">pod_dns.default.svc.cluster.local</property>
                        <property name="service-dns-timeout">10</property>
                    </properties>
                </discovery-strategy>
            </discovery-strategies>
        </join>
    </network>

Kubernetes StatefulSet. Bash scripts for hazelcast and orientdb-server-config files are mounted and executed (for setting update depending on the env value on each pod):

kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: orientdbservice
spec:
  serviceName: orientdbservice
  replicas: 3
  podManagementPolicy: Parallel
  selector:
    matchLabels:
      service: orientdb
      type: container-deployment
  template:
    metadata:
      labels:
        service: orientdb
        type: container-deployment
    spec:
      containers:
      - name: orientdbservice
        image: orientdb:2.2.36
        command: ["/bin/sh","-c", " cp /configs/* /orientdb/config/ ; chmod +x /orientdb/config/hazelcast_template.sh ; chmod +x /orientdb/config/server_config_template.sh ; sh /orientdb/config/hazelcast_template.sh ; sh /orientdb/config/server_config_template.sh ; /orientdb/bin/server.sh -Ddistributed=true" ]
        env:
        - name: ORIENTDB_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: orientdb-password
              key: password.txt
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        ports:
        - containerPort: 2424
          name: port-binary
        - containerPort: 2480
          name: port-http
        - containerPort: 5701
          name: hazelcast
        volumeMounts:
        - name: config
          mountPath: /orientdb/config
        - name: orientdb-config-template-hazelcast
          mountPath: /configs/hazelcast_template.sh
          subPath: hazelcast_template.sh
        - name: orientdb-config-template-server
          mountPath: /configs/server_config_template.sh
          subPath: server_config_template.sh
        - name: orientdb-config-distributed
          mountPath: /configs/default-distributed-db-config.json
          subPath: default-distributed-db-config.json
        - name: orientdb-databases
          mountPath: /orientdb/databases
        - name: orientdb-backup
          mountPath: /orientdb/backup
      volumes:
      - name: config
        emptyDir: {}
      - name: orientdb-config-template-hazelcast
        configMap:
          name: orientdb-configmap-template-hazelcast
      - name: orientdb-config-template-server
        configMap:
          name: orientdb-configmap-template-server
      - name: orientdb-config-distributed
        configMap:
          name: orientdb-configmap-distributed
  volumeClaimTemplates:
  - metadata:
      name: orientdb-databases
      labels:
        service: orientdb
        type: pv-claim
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 20Gi
  - metadata:
      name: orientdb-backup
      labels:
        service: orientdb
        type: pv-claim
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 10Gi
-- Cassie
kubernetes
orientdb
service-discovery

1 Answer

11/6/2018

The problem is in the Hazelcast0Kubernetes plugin configuration. First of all, it is necessary to update OrientDB version to latest - 3.0.10 with embedded newest Hazelcast version. Also, I have mounted hazelcast-kubernetes.jar dependency file directly into /orientdb/lib folder and it started to work properly. The problem was not about the config files but about the dependency setup for OrientDB.

-- Cassie
Source: StackOverflow