Deploy Neo4j cluster with kubernetes kubeadm

12/6/2018

Here is a guide to How to use Kubernetes to quickly deploy Neo4j clusters with minikube, so I follow these steps and it is just fine with minikube, but when I use kubeadm I facing problem.

I built a cluster with two VM, one of them is master and another a worker.
The worker joined the master successfully. In the master machine, I do

kubectl apply -f cores

Similar to the guide. And here is the cores.yamlfiles which it contains:

1) Statefulset.yaml:

apiVersion: "apps/v1beta1"
kind: StatefulSet
metadata:
  name: neo4j-core
spec:
  serviceName: neo4j
  replicas: 3
  template:
    metadata:
      labels:
        app: neo4j
        component: core
    spec:
      containers:
      - name: neo4j
        image: "neo4j:3.3.2-enterprise"
        imagePullPolicy: "IfNotPresent"
        env:
          - name: NEO4J_dbms_mode
            value: CORE
          - name: NUMBER_OF_CORES
            value: "3"
          - name: NEO4J_dbms_security_auth__enabled
            value: "false"
          - name: NEO4J_causal__clustering_discovery__type
            value: DNS
          - name: NEO4J_causal__clustering_initial__discovery__members
            value: "neo4j.default.svc.cluster.local:5000"
          - name: NEO4J_ACCEPT_LICENSE_AGREEMENT
            value: "yes"
        command:
          - "/bin/bash"
          - "-ecx"
          - |
            export NEO4J_dbms_connectors_default__advertised__address=$(hostname -f)
        export 
NEO4J_causal__clustering_discovery__advertised__address=$(hostname - 
f):5000
            export NEO4J_causal__clustering_transaction__advertised__address=$(hostname -f):6000
            export NEO4J_causal__clustering_raft__advertised__address=$(hostname -f):7000
            exec /docker-entrypoint.sh "neo4j"
        ports:
        - containerPort: 5000
          name: discovery
        - containerPort: 7000
          name: raft
        - containerPort: 6000
          name: tx
        - containerPort: 7474
          name: browser
        - containerPort: 7687
          name: bolt
        - containerPort: 6362
          name: backup
        securityContext:
          privileged: true
        volumeMounts:
        - name: datadir
          mountPath: "/data"
  volumeClaimTemplates:
    - metadata:
        name: datadir
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: "standard"
        resources:
          requests:
            storage: "10Gi"

2) dns.yaml:

apiVersion: v1
kind: Service
metadata:
  name: neo4j
  labels:
    app: neo4j
    component: core
spec:
  clusterIP: None
  ports:
    - port: 7474
      targetPort: 7474
      name: browser
    - port: 6362
      targetPort: 6362
      name: backup
  selector:
    app: neo4j
    component: core

And then when I do:

kubectl logs -l "app=neo4j"

Instead of seeing the following lines :

2017-09-13 09:41:39.562+0000 INFO  Remote interface available at
http://neo4j-core-2.neo4j.default.svc.cluster.local:7474/

Cores stuck in connecting to other members and says:

++ hostname -f
+ export NEO4J_dbms_connectors_default__advertised__address=neo4j-core-0.neo4j.default.svc.cluster.local
+ NEO4J_dbms_connectors_default__advertised__address=neo4j-core-0.neo4j.default.svc.cluster.local
++ hostname -f
+ export NEO4J_causal__clustering_discovery__advertised__address=neo4j-core-0.neo4j.default.svc.cluster.local:5000
+ NEO4J_causal__clustering_discovery__advertised__address=neo4j-core-0.neo4j.default.svc.cluster.local:5000
++ hostname -f
+ export NEO4J_causal__clustering_transaction__advertised__address=neo4j-core-0.neo4j.default.svc.cluster.local:6000
+ NEO4J_causal__clustering_transaction__advertised__address=neo4j-core-0.neo4j.default.svc.cluster.local:6000
++ hostname -f
+ export NEO4J_causal__clustering_raft__advertised__address=neo4j-core-0.neo4j.default.svc.cluster.local:7000
+ NEO4J_causal__clustering_raft__advertised__address=neo4j-core-0.neo4j.default.svc.cluster.local:7000
+ exec /docker-entrypoint.sh neo4j
Active database: graph.db
Directories in use:
  home:         /var/lib/neo4j
  config:       /var/lib/neo4j/conf
  logs:         /var/lib/neo4j/logs
  plugins:      /var/lib/neo4j/plugins
  import:       /var/lib/neo4j/import
  data:         /var/lib/neo4j/data
  certificates: /var/lib/neo4j/certificates
  run:          /var/lib/neo4j/run
Starting Neo4j.
2018-10-29 14:38:01.268+0000 INFO  ======== Neo4j 3.3.2 ========
2018-10-29 14:38:01.359+0000 INFO  Starting...
2018-10-29 14:38:03.680+0000 INFO  Bolt enabled on 0.0.0.0:7687.
2018-10-29 14:38:03.715+0000 INFO  Initiating metrics...
2018-10-29 14:38:04.021+0000 INFO  Resolved initial host 
'neo4j.neo4j.svc.cluster.local:5000' to []
2018-10-29 14:38:04.095+0000 INFO  My connection info: [
        Discovery:   listen=0.0.0.0:5000, advertised=neo4j-core-0.neo4j.default.svc.cluster.local:5000,
        Transaction: listen=0.0.0.0:6000, advertised=neo4j-core-0.neo4j.default.svc.cluster.local:6000, 
        Raft:        listen=0.0.0.0:7000, advertised=neo4j-core-0.neo4j.default.svc.cluster.local:7000, 
        Client Connector Addresses: bolt://neo4j-core-0.neo4j.default.svc.cluster.local:7687,http://neo4j-core-0.neo4j.default.svc.cluster.local:7474,https://neo4j-core-0.neo4j.default.svc.cluster.local:7473
]
    2018-10-29 14:38:04.095+0000 INFO  Discovering cluster with initial members: [neo4j.neo4j.svc.cluster.local:5000]
    2018-10-29 14:38:04.095+0000 INFO  Attempting to connect to the other cluster members before continuing...

It stays at last line and doesn't continue and even doesn't throw a failure error! What is wrong? Is it about using kubeadm instead of minikube?

-- Majid Rajabi
kubeadm
kubernetes
minikube
neo4j

1 Answer

12/6/2018

Not knowing too much about neo4j, my guess is that neo4j couldn't find the members themselves.

Since you are running 3 replicas im guessing you need to add the expected members fqdn in the initial__discovery__members variable.

-name: NEO4J_causal__clustering_initial__discovery__members 
-value: "neo4j-core-0.neo4j.default.svc.cluster.local:5000, neo4j-core-1.neo4j.default.svc.cluster.local:5000,neo4j-core-2.neo4j.default.svc.cluster.local:5000"

Would you be able to try that?

-- Bal Chua
Source: StackOverflow