My remotely working setup is as follows:
PersistentVolume
is mounted to a gcePersistentDisk
. Pod "Lagg" makes a claim on the entirety of the persistent disk. "Lagg" is a google containers volume-nfs image, which acts as the middleman between the ReadWriteOnce
volume and a NFS ReadWriteMany
that all of my other pods can access. Below is the Lagg NFS persistent volume YAML:
apiVersion: v1
kind: PersistentVolume
metadata:
name: lagg-volume
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteMany
nfs:
# kustomize does not add prefixes here, so they're placed ahead of time
server: test-lagg.test-project.svc.cluster.local
path: "/"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: lagg-claim
spec:
accessModes:
- ReadWriteMany
storageClassName: ""
resources:
requests:
storage: 5Gi
There is a second PersistentVolume
that mounts to the pod via NFS, that other pods can claim. One of those pods is "Digit" which you can see the volume defining part below:
spec:
template:
spec:
containers:
- name: digit
volumeMounts:
- name: lagg-connection
mountPath: "/cache"
volumes:
- name: lagg-connection
persistentVolumeClaim:
claimName: lagg-claim
Because I don't have a gcePersistentDisk
for local testing, my local version of this cluster instead uses another persistent volume called "Lagg-local" which simply takes the place of the gcePersistentDisk
, and looks like this:
apiVersion: v1
kind: PersistentVolume
metadata:
name: lagg-local-volume
labels:
type: local
spec:
storageClassName: manual
persistentVolumeReclaimPolicy: Delete
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
local:
path: /run/desktop/mnt/host/c/project/cache
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- docker-desktop
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: lagg-local-claim
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
When I try to run this locally, I only get one error, and it's in the Digit pod, using describe, it says:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 47s default-scheduler Successfully assigned test-project/test-digit-cff6bd9c6-gz2sn to docker-desktop
Warning FailedMount 11s (x7 over 43s) kubelet MountVolume.SetUp failed for volume "test-lagg-volume" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs test-lagg.test-project.svc.cluster.local:/ /var/lib/kubelet/pods/80f686cf-47bb-478b-a581-c179794e2182/volumes/kubernetes.io~nfs/test-lagg-volume
Output: mount.nfs: Failed to resolve server test-lagg.test-project.svc.cluster.local: Name or service not known
From what I can see, the pods simply can't contact the NFS server or possibly can't resolve the DNS. test-lagg exists and is running, and test-project is the namespace that both test-lagg (the service which points to the lagg NFS pods) and test-digit reside in. So I'm not entirely sure what is happening here.
I do believe the NFS server is working correctly, as a file "index.html" is created in the root of the volume that simply contains "Hello from NFS!"
The same error also happens if I use cpuguy83/nfs-server
image instead of google_containers/volume-nfs
A different error happens if I define the clusterIP
rather than the DNS name, stating it doesn't have permissions.
I also don't think there's an issue with the connection to the service because running nslookup on the digit pod returns this:
root@test-digit-7c6dc66659-q4trw:/var/www/static# nslookup test-lagg.test-project.svc.cluster.local
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: test-lagg.test-project.svc.cluster.local
Address: 10.105.85.125
The NFS pod itself also has the volume mounted correctly:
On GKE:
PS C:\Users\ral\Documents\Projects\Project\Kubernetes> kubectl exec next-lagg-69884bf49b-fn544 -- bash -c "findmnt /exports -o TARGET,SOURCE,FSTYPE"
TARGET SOURCE FSTYPE
/exports /dev/sdb ext4
On local:
PS C:\Users\ral\Documents\Projects\Project\Kubernetes> kubectl exec test-lagg-547cbb779-4qgbl -- bash -c "findmnt /exports -o TARGET,SOURCE,FSTYPE"
TARGET SOURCE FSTYPE
/exports C:\[/Project/cache] 9p
DNS resolution problems with google_containers/volume-nfs
, on non-GKE clusters, is a known issue:
Basically, NFS server does not support hostnames, only IPs.
Alternatively you could use csi-driver-nfs