K
Q

Neo4j shutdown when mounting volume via Kubernetes

9/5/2019

I have created a custom dockerfile for neo4j that imports necessary plugins. Next I created a deployment yaml which I am running on Kubernetes on Azure. The deployment works very well and I can reach the browser on the expected URL, which works great. However, the moment I mount a Volume, neo4j becomes unable to start with the below provided error stack. It seems that Azure is using the required directory which interferes with neo4j? However I am not sure how to fix this.

Any ideas?

Two notes:

  1. Note that the directory that is mount is empty, so there is no database yet.
  2. The persistent volume (claim) has been tested on a simple image like nginx, and works fine.

Find below the error stack and deployment yaml.

My demployment yaml

apiVersion: v1
kind: Service
metadata:
  name: myservice
spec:
  selector:
    app: mygraph
  ports:
    - name: bolt
      protocol: TCP
      port: 7687
      targetPort: 7687
      nodePort: 30076
    - name: http
      protocol: TCP
      port: 7474
      targetPort: 7474
      nodePort: 30074
    - name: https
      protocol: TCP
      port: 7473
      targetPort: 7473
      nodePort: 30073
  type: LoadBalancer
  loadBalancerIP: 40.118.172.46
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mygraph-deployment
  labels:
    app: mygraph
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mygraph
  template:
    metadata:
      labels:
        app: mygraph
    spec:
      containers:
      - name: mygraph
        image: mycontainer.azurecr.io/mygraph:dev
        securityContext:
          privileged: true
        ports:
        - containerPort: 7687
        - containerPort: 7474
        - containerPort: 7473
        env:
          - name: NEO4J_dbms_jvm_additional
            value: "-Xms512m -Xmx6g"
        # When adding the below lines the deployment failes. Without it, it works great!
        volumeMounts:
        - name: neo4jdata
          mountPath: /data
      volumes:
        - name: neo4jdata
          persistentVolumeClaim:
            claimName: azurefile

And the corresponding error stack

2019-09-05 07:02:12.522+0000 INFO ======== Neo4j 3.3.9 ========

2019-09-05 07:02:12.560+0000 INFO Starting...

2019-09-05 07:02:14.122+0000 INFO Bolt enabled on 0.0.0.0:7687.

2019-09-05 07:02:14.274+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@149dd36b' was successfully initialized, but failed to start. Please see the attached cause exception "Unable to create directory path [/var/lib/neo4j/data/databases/graph.db] for Neo4j store.". Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@149dd36b' was successfully initialized, but failed to start. Please see the attached cause exception "Unable to create directory path [/var/lib/neo4j/data/databases/graph.db] for Neo4j store.".

org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@149dd36b' was successfully initialized, but failed to start. Please see the attached cause exception "Unable to create directory path [/var/lib/neo4j/data/databases/graph.db] for Neo4j store.".

at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:68)

at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:220)

at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:111)

at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:79)

at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:32)

Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase@149dd36b' was successfully initialized, but failed to start. Please see the attached cause exception "Unable to create directory path [/var/lib/neo4j/data/databases/graph.db] for Neo4j store.".

at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:466)

at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)

at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:212)

... 3 more

Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory, /var/lib/neo4j/data/databases/graph.db

at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:211)

at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:126)

at org.neo4j.server.CommunityNeoServer.lambda$static$0(CommunityNeoServer.java:58)

at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:88)

at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:445)

... 5 more

Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.internal.locker.StoreLockerLifecycleAdapter@134d26af' was successfully initialized, but failed to start. Please see the attached cause exception "Unable to create directory path [/var/lib/neo4j/data/databases/graph.db] for Neo4j store.".

at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:466)

at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)

at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:207)

... 9 more

Caused by: org.neo4j.kernel.StoreLockException: Unable to create path for store dir: /var/lib/neo4j/data/databases/graph.db. Please ensure no other process is using this database, and that the directory is writable (required even for read-only access)

at org.neo4j.kernel.internal.locker.StoreLocker.storeLockException(StoreLocker.java:116)

at org.neo4j.kernel.internal.locker.StoreLocker.checkLock(StoreLocker.java:79)

at org.neo4j.kernel.internal.locker.GlobalStoreLocker.checkLock(GlobalStoreLocker.java:60)

at org.neo4j.kernel.internal.locker.StoreLockerLifecycleAdapter.start(StoreLockerLifecycleAdapter.java:36)

at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:445)

... 11 more

Caused by: java.io.IOException: Unable to create directory path [/var/lib/neo4j/data/databases/graph.db] for Neo4j store.

at org.neo4j.io.fs.DefaultFileSystemAbstraction.mkdirs(DefaultFileSystemAbstraction.java:124)

at org.neo4j.kernel.internal.locker.StoreLocker.checkLock(StoreLocker.java:73)

... 14 more

2019-09-05 07:02:14.317+0000 INFO Neo4j Server shutdown initiated by request
-- N Meibergen
azure
kubernetes
neo4j
persistent-volumes

Similar Questions

Not able to access kubernetes cluster service on a taint node
how to make ingress nginx return 200 on each request
Master not pods in NotReady status
error parsing templates/deployment.yaml: json: line 1: invalid character '{' looking for beginning of object key string
Minio bucket failed to copy, Please reduce your request

1 Answer

9/9/2019

Found out that it was an issue of rights of writing data to Azure. In particular in the setup of the StorageClass you can set additional mountOptions. One of those is the uid which should be set to 100 - the user id used for neo4j.

Example of the StorageClass yaml:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azurefile-neo4j
provisioner: kubernetes.io/azure-file
reclaimPolicy: Retain
mountOptions:
  - dir_mode=0755
  - file_mode=0755
  - uid=100 # Allow write access for neo4j
  - gid=1000
parameters:
  skuName: Standard_LRS

It is curious though that the file /data/dbms was written successful before, while /data/databases could not be written. Seems that dbms and databases are created by different users? But that would be a separate question.

-- N Meibergen
Source: StackOverflow