I am trying to build an OrientDB cluster which consists of 3 nodes with Kubernetes (in Minikube). I am using StatefulSet with all the cluster configurations mounted as volumes. The problem is when I check all the pods' logs it looks like each pod creates its own cluster instead of joining a single cluster. What could be the cause of it?
Hazelcast.xml config looks like this:
<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.3.xsd"
xmlns="http://www.hazelcast.com/schema/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<properties>
<property name="hazelcast.discovery.enabled">true</property>
</properties>
<network>
<join>
<!-- deactivate normal discovery -->
<multicast enabled="false"/>
<tcp-ip enabled="false"/>
<!-- activate the Kubernetes plugin -->
<discovery-strategies>
<discovery-strategy enabled="true"
class="com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy">
<properties>
<property name="service-dns">orientdbservice.default.svc.cluster.local</property>
<property name="service-dns-timeout">10</property>
</properties>
</discovery-strategy>
</discovery-strategies>
</join>
</network>
</hazelcast>
I am using service dns name as one of the approaches to use Hazelcast plugin with Kubernetes.
default-distributed-db-config.json file:
{
"autoDeploy": true,
"hotAlignment": false,
"executionMode": "undefined",
"readQuorum": 1,
"writeQuorum": "majority",
"readYourWrites": true,
"newNodeStrategy": "static",
"servers": {
"*": "master"
},
"clusters":{
"internal": {
},
"index": {
},
"*": {
"servers" : [ "<NEW_NODE" ]
}
}
}
Listeners part from orientdb-server-config.xml:
<listeners>
<listener protocol="binary" socket="default" port-range="2424-2430" ip-address="0.0.0.0"/>
<listener protocol="http" socket="default" port-range="2480-2490" ip-address="0.0.0.0">
<commands>
<command implementation="com.orientechnologies.orient.server.network.protocol.http.command.get.OServerCommandGetStaticContent" pattern="GET|www GET|studio/ GET| GET|*.htm GET|*.html GET|*.xml GET|*.jpeg GET|*.jpg GET|*.png GET|*.gif GET|*.js GET|*.css GET|*.swf GET|*.ico GET|*.txt GET|*.otf GET|*.pjs GET|*.svg GET|*.json GET|*.woff GET|*.woff2 GET|*.ttf GET|*.svgz" stateful="false">
<parameters>
<entry value="Cache-Control: no-cache, no-store, max-age=0, must-revalidate\r\nPragma: no-cache" name="http.cache:*.htm *.html"/>
<entry value="Cache-Control: max-age=120" name="http.cache:default"/>
</parameters>
</command>
</commands>
<parameters>
<parameter value="utf-8" name="network.http.charset"/>
<parameter value="true" name="network.http.jsonResponseError"/>
<parameter value="Access-Control-Allow-Origin:*;Access-Control-Allow-Credentials: true" name="network.http.additionalResponseHeaders"/>
</parameters>
</listener>
</listeners>
StatefulSet I have defined:
kind: StatefulSet
apiVersion: apps/v1
metadata:
name: orientdbservice
spec:
serviceName: orientdbservice
replicas: 3
selector:
matchLabels:
service: orientdb
type: container-deployment
template:
metadata:
labels:
service: orientdb
type: container-deployment
spec:
containers:
- name: orientdbservice
image: orientdb:2.2.36
command: ["/bin/sh","-c", "cp /configs/* /orientdb/config/ ; /orientdb/bin/server.sh -Ddistributed=true" ]
env:
- name: ORIENTDB_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: orientdb-password
key: password.txt
ports:
- containerPort: 2424
name: port-binary
- containerPort: 2480
name: port-http
- containerPort: 5701
name: hazelcast
volumeMounts:
- name: config
mountPath: /orientdb/config
- name: orientdb-config-backups
mountPath: /configs/backups.json
subPath: backups.json
- name: orientdb-config-events
mountPath: /configs/events.json
subPath: events.json
- name: orientdb-config-distributed
mountPath: /configs/default-distributed-db-config.json
subPath: default-distributed-db-config.json
- name: orientdb-config-hazelcast
mountPath: /configs/hazelcast.xml
subPath: hazelcast.xml
- name: orientdb-config-server
mountPath: /configs/orientdb-server-config.xml
subPath: orientdb-server-config.xml
- name: orientdb-config-client-logs
mountPath: /configs/orientdb-client-log.properties
subPath: orientdb-client-log.properties
- name: orientdb-config-server-logs
mountPath: /configs/orientdb-server-log.properties
subPath: orientdb-server-log.properties
- name: orientdb-config-plugin
mountPath: /configs/pom.xml
subPath: pom.xml
- name: orientdb-databases
mountPath: /orientdb/databases
- name: orientdb-backup
mountPath: /orientdb/backup
- name: orientdb-data
mountPath: /orientdb/bin/data
volumes:
- name: config
emptyDir: {}
- name: orientdb-config-backups
configMap:
name: orientdb-configmap-backups
- name: orientdb-config-events
configMap:
name: orientdb-configmap-events
- name: orientdb-config-distributed
configMap:
name: orientdb-configmap-distributed
- name: orientdb-config-hazelcast
configMap:
name: orientdb-configmap-hazelcast
- name: orientdb-config-server
configMap:
name: orientdb-configmap-server
- name: orientdb-config-client-logs
configMap:
name: orientdb-configmap-client-logs
- name: orientdb-config-server-logs
configMap:
name: orientdb-configmap-server-logs
- name: orientdb-config-plugin
configMap:
name: orientdb-configmap-plugin
- name: orientdb-data
hostPath:
path: /import_data
type: Directory
volumeClaimTemplates:
- metadata:
name: orientdb-databases
labels:
service: orientdb
type: pv-claim
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 20Gi
- metadata:
name: orientdb-backup
labels:
service: orientdb
type: pv-claim
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 10Gi
Headkess service looks like this:
kind: Service
apiVersion: v1
metadata:
name: orientdbservice
labels:
service: orientdb
type: headless-service
spec:
type: ClusterIP
clusterIP: None
selector:
service: orientdb
type: container-deployment
ports:
- port: 2424
name: port-binary
- port: 2480
name: port-http
- name: hazelcast
port: 5701
For Hazelcast members to discover themselves, you also need to:
containerPort
for Hazelcast (port 5701
)5701
, in case of using DNS Lookup Discovery, the service needs to be type: ClusterIP
and clusterIP: None
)You can find more information about the configuration in the following sources: