Pod can't mount to NFS pod on Docker Desktop local test environment

10/16/2021

My remotely working setup is as follows:

PersistentVolume is mounted to a gcePersistentDisk. Pod "Lagg" makes a claim on the entirety of the persistent disk. "Lagg" is a google containers volume-nfs image, which acts as the middleman between the ReadWriteOnce volume and a NFS ReadWriteMany that all of my other pods can access. Below is the Lagg NFS persistent volume YAML:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: lagg-volume
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  nfs:
    # kustomize does not add prefixes here, so they're placed ahead of time
    server: test-lagg.test-project.svc.cluster.local
    path: "/"

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: lagg-claim
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  resources:
    requests:
      storage: 5Gi

There is a second PersistentVolume that mounts to the pod via NFS, that other pods can claim. One of those pods is "Digit" which you can see the volume defining part below:

spec:
  template:
    spec:
      containers:
      - name: digit
        volumeMounts:
          - name: lagg-connection
            mountPath: "/cache"
      volumes:
      - name: lagg-connection
        persistentVolumeClaim:
          claimName: lagg-claim

Because I don't have a gcePersistentDisk for local testing, my local version of this cluster instead uses another persistent volume called "Lagg-local" which simply takes the place of the gcePersistentDisk, and looks like this:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: lagg-local-volume
  labels:
    type: local
spec:
  storageClassName: manual
  persistentVolumeReclaimPolicy: Delete
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  local:
    path: /run/desktop/mnt/host/c/project/cache
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - docker-desktop
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: lagg-local-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

When I try to run this locally, I only get one error, and it's in the Digit pod, using describe, it says:

Events:
  Type     Reason       Age                From               Message
  ----     ------       ----               ----               -------
  Normal   Scheduled    47s                default-scheduler  Successfully assigned test-project/test-digit-cff6bd9c6-gz2sn to docker-desktop
  Warning  FailedMount  11s (x7 over 43s)  kubelet            MountVolume.SetUp failed for volume "test-lagg-volume" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs test-lagg.test-project.svc.cluster.local:/ /var/lib/kubelet/pods/80f686cf-47bb-478b-a581-c179794e2182/volumes/kubernetes.io~nfs/test-lagg-volume
Output: mount.nfs: Failed to resolve server test-lagg.test-project.svc.cluster.local: Name or service not known

From what I can see, the pods simply can't contact the NFS server or possibly can't resolve the DNS. test-lagg exists and is running, and test-project is the namespace that both test-lagg (the service which points to the lagg NFS pods) and test-digit reside in. So I'm not entirely sure what is happening here.

I do believe the NFS server is working correctly, as a file "index.html" is created in the root of the volume that simply contains "Hello from NFS!"

The same error also happens if I use cpuguy83/nfs-server image instead of google_containers/volume-nfs

A different error happens if I define the clusterIP rather than the DNS name, stating it doesn't have permissions.

I also don't think there's an issue with the connection to the service because running nslookup on the digit pod returns this:

root@test-digit-7c6dc66659-q4trw:/var/www/static# nslookup test-lagg.test-project.svc.cluster.local
Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   test-lagg.test-project.svc.cluster.local
Address: 10.105.85.125

The NFS pod itself also has the volume mounted correctly:

On GKE:
PS C:\Users\ral\Documents\Projects\Project\Kubernetes> kubectl exec next-lagg-69884bf49b-fn544 -- bash -c "findmnt /exports -o TARGET,SOURCE,FSTYPE"
TARGET   SOURCE   FSTYPE
/exports /dev/sdb ext4

On local:
PS C:\Users\ral\Documents\Projects\Project\Kubernetes> kubectl exec test-lagg-547cbb779-4qgbl -- bash -c "findmnt /exports -o TARGET,SOURCE,FSTYPE"
TARGET   SOURCE             FSTYPE
/exports C:\[/Project/cache] 9p
-- Ral
docker-desktop
kubernetes
kubernetes-pod
persistent-volumes

1 Answer

10/19/2021

DNS resolution problems with google_containers/volume-nfs, on non-GKE clusters, is a known issue:

Basically, NFS server does not support hostnames, only IPs.

Alternatively you could use csi-driver-nfs

-- p10l
Source: StackOverflow