K
Q

HDFS Namenode format issue with AWS EBS in EKS cluster

5/9/2020

I have EKS cluster with EBS storage class/volume. I have elasticsearch cluster running fine with this EBS storage (as persistent volume/pvc). I am trying to deploy hdfs namenode image (bde2020/hadoop-namenode) using statefulset, but it gives me below error always:

2020-05-09 08:59:02,400 INFO util.GSet: capacity      = 2^15 = 32768 entries
2020-05-09 08:59:02,415 INFO common.Storage: Lock on /hadoop/dfs/name/in_use.lock acquired by nodename 87@hdfs-name-0.hdfs-name.pulse.svc.cluster.local
2020-05-09 08:59:02,417 WARN namenode.FSNamesystem: Encountered exception loading fsimage
java.io.IOException: NameNode is not formatted.
    at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:252)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1105)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:720)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:648)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:710)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:953)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:926)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1692)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1759)

I checked run.sh of this iameg and it does seem to be formatting namenode if dir is empty. But this does not work in may case (with EBS as PVC). Any help would be very appreciated.

My deployment yml is:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hdfs-name
  labels:
    component: hdfs-name
spec:
  serviceName: hdfs-name
  replicas: 1
  selector:
    matchLabels:
      component: hdfs-name
  template:
    metadata:
      labels:
        component: hdfs-name
    spec:
      containers:
      - name: hdfs-name
        image: bde2020/hadoop-namenode
        env:
        - name: CLUSTER_NAME
          value: hdfs-k8s
        ports:
        - containerPort: 8020
          name: nn-rpc
        - containerPort: 50070
          name: nn-web
        volumeMounts:
        - name: hdfs-name-pv-claim
          mountPath: /hadoop/dfs/name 
  volumeClaimTemplates:
  - metadata:
      name: hdfs-name-pv-claim
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: ebs
      resources:
        requests:
          storage: 1Gi
-- NumeroUno
hdfs
kubernetes

Similar Questions

Redirect logs of pods in my k8s to a file with pod name
Is there any way to represent POD CPU usage in terms of CPU cores using prometheus metrics
k8s job completed but the pod shows "OOMKILLED"
Enabling SSL on GKE endpoints not working correctly

1 Answer

5/9/2020

With ebs storage class, lost+found folder is created automatically. Due to this, namenode format does not occur.
Having initcontainer to delete the lost+found folder seems to work.

initContainers:
  - name: delete-lost-found
    image: busybox
    command: ["sh", "-c", "rm -rf /hadoop/dfs/name/lost+found"]
    volumeMounts:
    - name: hdfs-name-pv-claim
      mountPath: /hadoop/dfs/name
-- NumeroUno
Source: StackOverflow