K
Q

Unable to attach or mount volumes: unmounted volumes

1/27/2021

i deplyed my application on kubernetes but have been getting this error: 

**MountVolume.SetUp failed for volume "airflow-volume" : mount failed: mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/4a3c3d0b-b7e8-49bc-8a78-5a8bdc932eca/volumes/kubernetes.io~glusterfs/airflow-volume --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=10.0.2.107:10.0.2.24,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/airflow-volume/worker-844c9db787-vprt8-glusterfs.log,log-level=ERROR 10.0.2.107:/airflow /var/lib/kubelet/pods/4a3c3d0b-b7e8-49bc-8a78-5a8bdc932eca/volumes/kubernetes.io~glusterfs/airflow-volume Output: Running scope as unit run-22059.scope. mount: /var/lib/kubelet/pods/4a3c3d0b-b7e8-49bc-8a78-5a8bdc932eca/volumes/kubernetes.io~glusterfs/airflow-volume: unknown filesystem type 'glusterfs'. , the following error information was pulled from the glusterfs log to help diagnose this issue: could not open log file for pod worker-844c9db787-vprt8**

AND

**Unable to attach or mount volumes: unmounted volumes=[airflow-volume], unattached volumes=[airflow-volume default-token-s6pvd]: timed out waiting for the condition**

Any suggestions? 

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: airflow
spec:
  replicas: 1
  selector:
    matchLabels:
      tier: web
  template:
    metadata:
      labels:
        app: airflow
        tier: web
    spec:
      imagePullSecrets:
        - name: peeriqregistrykey
      restartPolicy: Always
      containers:
        # Airflow Webserver Container
        - name: web
          image: peeriq/data_availability_service:airflow-metadata-cutover
          volumeMounts:
            - mountPath: /usr/local/airflow
              name: airflow-volume
          envFrom:
            - configMapRef:
                name: airflow-config
          env:
            - name: VAULT_ADDR
              valueFrom:
                secretKeyRef:
                  name: vault-credentials
                  key: VAULT_ADDR
            - name: VAULT_TOKEN
              valueFrom:
                secretKeyRef:
                  name: vault-credentials
                  key: VAULT_TOKEN
            - name: DJANGO_AUTH_USER
              valueFrom:
                secretKeyRef:
                  name: django-auth
                  key: DJANGO_AUTH_USER
            - name: DJANGO_AUTH_PASS
              valueFrom:
                secretKeyRef:
                  name: django-auth
                  key: DJANGO_AUTH_PASS
            - name: FERNET_KEY
              valueFrom:
                secretKeyRef:
                  name: airflow-secrets
                  key: FERNET_KEY
            - name: POSTGRES_SERVICE_HOST
              valueFrom:
                secretKeyRef:
                  name: rds-postgres
                  key: POSTGRES_SERVICE_HOST
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: rds-postgres
                  key: POSTGRES_PASSWORD
          ports:
            - name: web
              containerPort: 8080
          args: ["webserver"]
        # Airflow Scheduler Container
        - name: scheduler
          image: peeriq/data_availability_service:airflow-metadata-cutover
          volumeMounts:
            - mountPath: /usr/local/airflow
              name: airflow-volume
          envFrom:
            - configMapRef:
                name: airflow-config
          env:
            - name: AWS_DEFAULT_REGION
              value: us-east-1
            - name: ETL_AWS_ACCOUNT_NUMBER
              valueFrom:
                secretKeyRef:
                  name: aws-creds
                  key: ETL_AWS_ACCOUNT_NUMBER
            - name: VAULT_ADDR
              valueFrom:
                secretKeyRef:
                  name: vault-credentials
                  key: VAULT_ADDR
            - name: VAULT_TOKEN
              valueFrom:
                secretKeyRef:
                  name: vault-credentials
                  key: VAULT_TOKEN
            - name: DJANGO_AUTH_USER
              valueFrom:
                secretKeyRef:
                  name: django-auth
                  key: DJANGO_AUTH_USER
            - name: DJANGO_AUTH_PASS
              valueFrom:
                secretKeyRef:
                  name: django-auth
                  key: DJANGO_AUTH_PASS
            - name: FERNET_KEY
              valueFrom:
                secretKeyRef:
                  name: airflow-secrets
                  key: FERNET_KEY
            - name: POSTGRES_SERVICE_HOST
              valueFrom:
                secretKeyRef:
                  name: rds-postgres
                  key: POSTGRES_SERVICE_HOST
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: rds-postgres
                  key: POSTGRES_PASSWORD
          args: ["scheduler"]
      volumes:
        - name: airflow-volume
          # This GlusterFS volume must already exist.
          glusterfs:
            endpoints: glusterfs-cluster
            path: /airflow
            readOnly: false
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: flower
  namespace: airflow
spec:
  replicas: 1
  selector:
    matchLabels:
      tier: flower
  template:
    metadata:
      labels:
        app: airflow
        tier: flower
    spec:
      imagePullSecrets:
        - name: peeriqregistrykey
      restartPolicy: Always
      containers:
        - name: flower
          image: peeriq/data_availability_service:airflow-metadata-cutover
          volumeMounts:
            - mountPath: /usr/local/airflow
              name: airflow-volume
          envFrom:
            - configMapRef:
                name: airflow-config
          env:
            # To prevent the error: ValueError: invalid literal for int() with base 10: 'tcp://10.0.0.83:5555'
            - name: FLOWER_PORT
              value: "5555"
            - name: DJANGO_AUTH_USER
              valueFrom:
                secretKeyRef:
                  name: django-auth
                  key: DJANGO_AUTH_USER
            - name: DJANGO_AUTH_PASS
              valueFrom:
                secretKeyRef:
                  name: django-auth
                  key: DJANGO_AUTH_PASS
            - name: POSTGRES_SERVICE_HOST
              valueFrom:
                secretKeyRef:
                  name: rds-postgres
                  key: POSTGRES_SERVICE_HOST
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: rds-postgres
                  key: POSTGRES_PASSWORD
          ports:
            - name: flower
              containerPort: 5555
          args: ["flower"]
      volumes:
        - name: airflow-volume
          # This GlusterFS volume must already exist.
          glusterfs:
            endpoints: glusterfs-cluster
            path: /airflow
            readOnly: false
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: worker
  namespace: airflow
spec:
  replicas: 1
  selector:
    matchLabels:
      tier: worker
  template:
    metadata:
      labels:
        app: airflow
        tier: worker
    spec:
      imagePullSecrets:
        - name: peeriqregistrykey
      restartPolicy: Always
      containers:
        - name: worker
          image: peeriq/data_availability_service:airflow-metadata-cutover
          volumeMounts:
            - mountPath: /usr/local/airflow
              name: airflow-volume
          envFrom:
            - configMapRef:
                name: airflow-config
          env:
            - name: AWS_DEFAULT_REGION
              value: us-east-1
            - name: ETL_AWS_ACCOUNT_NUMBER
              valueFrom:
                secretKeyRef:
                  name: aws-creds
                  key: ETL_AWS_ACCOUNT_NUMBER
            - name: VAULT_ADDR
              valueFrom:
                secretKeyRef:
                  name: vault-credentials
                  key: VAULT_ADDR
            - name: VAULT_TOKEN
              valueFrom:
                secretKeyRef:
                  name: vault-credentials
                  key: VAULT_TOKEN
            - name: DJANGO_AUTH_USER
              valueFrom:
                secretKeyRef:
                  name: django-auth
                  key: DJANGO_AUTH_USER
            - name: DJANGO_AUTH_PASS
              valueFrom:
                secretKeyRef:
                  name: django-auth
                  key: DJANGO_AUTH_PASS
            - name: FERNET_KEY
              valueFrom:
                secretKeyRef:
                  name: airflow-secrets
                  key: FERNET_KEY
            - name: POSTGRES_SERVICE_HOST
              valueFrom:
                secretKeyRef:
                  name: rds-postgres
                  key: POSTGRES_SERVICE_HOST
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: rds-postgres
                  key: POSTGRES_PASSWORD
          args: ["worker"]
      volumes:
        - name: airflow-volume
          # This GlusterFS volume must already exist.
          glusterfs:
            endpoints: glusterfs-cluster
            path: /airflow
            readOnly: false
-- Gaurav Jaitly
amazon-eks
kubernetes

Similar Questions

Subscribe cloud pub/sub topic from app running on Kubernetes
Website images work locally on Minikube but not remotely on GKE
How to simulate Power Failure In Kubernetes
Airflow - Conditional retry in KubernetesPodOperator
Local singularity image in snakemake workflow on AWS eks with --kubernetes option

3 Answers

1/27/2021

The part of the message unknown filesystem type 'glusterfs' can mean that there is something wrong with your volume definition or a storage class if you use it. But this is a guess.

-- Alexey S.
Source: StackOverflow
11/10/2021

I had the same error, and the reason in my kubernetes was that the nfs server was unavailable. After starting the nfs server, it was solved.

-- Alex
Source: StackOverflow
1/27/2021

You must install package glusterfs-fuse on your Kubernetes nodes, otherwise it won't be able to mount glusterfs volumes.

-- Vasili Angapov
Source: StackOverflow