I have 3-node kubernetes, host names are host_1, host_2, host_3.
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
host_1 Ready master 134d v1.10.1
host_2 Ready <none> 134d v1.10.1
host_3 Ready <none> 134d v1.10.1
I have defined 3 local persistent volumes of size 100M, mapped to a local directory on each node. I used the following descriptor 3 times where <hostname>
is one of: host_1, host_2, host_3:
apiVersion: v1
kind: PersistentVolume
metadata:
name: test-volume-<hostname>
spec:
capacity:
storage: 100M
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-storage
local:
path: /opt/jnetx/volumes/test-volume
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- <hostname>
After applying three such yamls, I have the following:
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
test-volume-host_1 100M RWO Delete Available local-storage 58m
test-volume-host_2 100M RWO Delete Available local-storage 58m
test-volume-host_3 100M RWO Delete Available local-storage 58m
Now, I have a very simple container that writes to a file. The file should be located on the local persistent volume. I deploy it as a statefulset with 1 replica and map volumes via statefulset's volumeClaimTemplates:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: filewriter
spec:
serviceName: filewriter
...
replicas: 1
template:
spec:
containers:
- name: filewriter
...
volumeMounts:
- mountPath: /test/data
name: fw-pv-claim
volumeClaimTemplates:
- metadata:
name: fw-pv-claim
spec:
accessModes:
- ReadWriteOnce
storageClassName: local-storage
resources:
requests:
storage: 100M
The volume claim seems to have been created ok and bound to pv on the first host:
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
test-volume-host_1 100M RWO Delete Bound default/fw-pv-claim-filewriter-0 local-storage 1m
test-volume-host_2 100M RWO Delete Available local-storage 1h
test-volume-host_3 100M RWO Delete Available local-storage 1h
But, the pod hangs in Pending state:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
filewriter-0 0/1 Pending 0 4s
If we describe, we can see the following errors:
$ kubectl describe pod filewriter-0
Name: filewriter-0
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2s (x8 over 1m) default-scheduler 0/3 nodes are available: 1 node(s) had taints that the pod didn't tolerate, 2 node(s) had volume node affinity conflict.
Can you help me figure out what is wrong? Why can't it just create the pod?
It seems that the one node where the PV is available has a taint that your StatefulSet does not have toleration for.