Nodes do not find each other OrientDB Kubernetes cluster

10/18/2018

I am trying to build an OrientDB cluster which consists of 3 nodes with Kubernetes (in Minikube). I am using StatefulSet with all the cluster configurations mounted as volumes. The problem is when I check all the pods' logs it looks like each pod creates its own cluster instead of joining a single cluster. What could be the cause of it?

Hazelcast.xml config looks like this:

<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.3.xsd"
           xmlns="http://www.hazelcast.com/schema/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <properties>
    <property name="hazelcast.discovery.enabled">true</property>
  </properties>
  <network>
    <join>
      <!-- deactivate normal discovery -->
      <multicast enabled="false"/>
      <tcp-ip enabled="false"/>
      <!-- activate the Kubernetes plugin -->
      <discovery-strategies>
        <discovery-strategy enabled="true"
                            class="com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy">
          <properties>
            <property name="service-dns">orientdbservice.default.svc.cluster.local</property>
            <property name="service-dns-timeout">10</property>
          </properties>
        </discovery-strategy>
      </discovery-strategies>
    </join>
  </network>
</hazelcast> 

I am using service dns name as one of the approaches to use Hazelcast plugin with Kubernetes.

default-distributed-db-config.json file:

{
    "autoDeploy": true,
    "hotAlignment": false, 
    "executionMode": "undefined",
    "readQuorum": 1,
    "writeQuorum": "majority",
    "readYourWrites": true,
    "newNodeStrategy": "static",
    "servers": {
        "*": "master"
    },
    "clusters":{
        "internal": {
        },
        "index": {
    },
        "*": { 
      "servers" : [ "<NEW_NODE" ] 
    } 
    }
}   

Listeners part from orientdb-server-config.xml:

<listeners>
    <listener protocol="binary" socket="default" port-range="2424-2430" ip-address="0.0.0.0"/>
    <listener protocol="http" socket="default" port-range="2480-2490" ip-address="0.0.0.0">
        <commands>
            <command implementation="com.orientechnologies.orient.server.network.protocol.http.command.get.OServerCommandGetStaticContent" pattern="GET|www GET|studio/ GET| GET|*.htm GET|*.html GET|*.xml GET|*.jpeg GET|*.jpg GET|*.png GET|*.gif GET|*.js GET|*.css GET|*.swf GET|*.ico GET|*.txt GET|*.otf GET|*.pjs GET|*.svg GET|*.json GET|*.woff GET|*.woff2 GET|*.ttf GET|*.svgz" stateful="false">
                <parameters>
                    <entry value="Cache-Control: no-cache, no-store, max-age=0, must-revalidate\r\nPragma: no-cache" name="http.cache:*.htm *.html"/>
                    <entry value="Cache-Control: max-age=120" name="http.cache:default"/>
                </parameters>
            </command>
        </commands>
        <parameters>
            <parameter value="utf-8" name="network.http.charset"/>
            <parameter value="true" name="network.http.jsonResponseError"/>
            <parameter value="Access-Control-Allow-Origin:*;Access-Control-Allow-Credentials: true" name="network.http.additionalResponseHeaders"/>
        </parameters>
    </listener>
</listeners>

StatefulSet I have defined:

kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: orientdbservice
spec:
  serviceName: orientdbservice
  replicas: 3
  selector:
    matchLabels:
      service: orientdb
      type: container-deployment
  template:
    metadata:
      labels:
        service: orientdb
        type: container-deployment
    spec:
      containers:
      - name: orientdbservice
        image: orientdb:2.2.36
        command: ["/bin/sh","-c", "cp /configs/* /orientdb/config/ ; /orientdb/bin/server.sh -Ddistributed=true" ]
        env:
        - name: ORIENTDB_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: orientdb-password
              key: password.txt
        ports:
        - containerPort: 2424
          name: port-binary
        - containerPort: 2480
          name: port-http
        - containerPort: 5701
          name: hazelcast
        volumeMounts:
        - name: config
          mountPath: /orientdb/config
        - name: orientdb-config-backups
          mountPath: /configs/backups.json
          subPath: backups.json
        - name: orientdb-config-events
          mountPath: /configs/events.json
          subPath: events.json
        - name: orientdb-config-distributed
          mountPath: /configs/default-distributed-db-config.json
          subPath: default-distributed-db-config.json
        - name: orientdb-config-hazelcast
          mountPath: /configs/hazelcast.xml
          subPath: hazelcast.xml
        - name: orientdb-config-server
          mountPath: /configs/orientdb-server-config.xml
          subPath: orientdb-server-config.xml
        - name: orientdb-config-client-logs
          mountPath: /configs/orientdb-client-log.properties
          subPath: orientdb-client-log.properties
        - name: orientdb-config-server-logs
          mountPath: /configs/orientdb-server-log.properties
          subPath: orientdb-server-log.properties
        - name: orientdb-config-plugin
          mountPath: /configs/pom.xml
          subPath: pom.xml
        - name: orientdb-databases
          mountPath: /orientdb/databases
        - name: orientdb-backup
          mountPath: /orientdb/backup
        - name: orientdb-data
          mountPath: /orientdb/bin/data
      volumes:
      - name: config
        emptyDir: {}
      - name: orientdb-config-backups
        configMap:
          name: orientdb-configmap-backups
      - name: orientdb-config-events
        configMap:
          name: orientdb-configmap-events
      - name: orientdb-config-distributed
        configMap:
          name: orientdb-configmap-distributed
      - name: orientdb-config-hazelcast
        configMap:
          name: orientdb-configmap-hazelcast
      - name: orientdb-config-server
        configMap:
          name: orientdb-configmap-server
      - name: orientdb-config-client-logs
        configMap:
          name: orientdb-configmap-client-logs
      - name: orientdb-config-server-logs
        configMap:
          name: orientdb-configmap-server-logs
      - name: orientdb-config-plugin
        configMap:
          name: orientdb-configmap-plugin
      - name: orientdb-data
        hostPath:
          path: /import_data
          type: Directory
  volumeClaimTemplates:
  - metadata:
      name: orientdb-databases
      labels:
        service: orientdb
        type: pv-claim
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 20Gi
  - metadata:
      name: orientdb-backup
      labels:
        service: orientdb
        type: pv-claim
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 10Gi

Headkess service looks like this:

kind: Service
apiVersion: v1
metadata:
  name: orientdbservice
  labels:
    service: orientdb
    type: headless-service
spec:
  type: ClusterIP
  clusterIP: None
  selector:
    service: orientdb
    type: container-deployment
  ports:
  - port: 2424
    name: port-binary
  - port: 2480
    name: port-http
  - name: hazelcast
    port: 5701
-- Cassie
hazelcast
kubernetes
minikube
orientdb

1 Answer

10/22/2018

For Hazelcast members to discover themselves, you also need to:

  • specify containerPort for Hazelcast (port 5701)
  • create a Kubernetes Service for Hazelcast (port 5701, in case of using DNS Lookup Discovery, the service needs to be type: ClusterIP and clusterIP: None)

You can find more information about the configuration in the following sources:

-- RafaƂ Leszko
Source: StackOverflow