How to manage pod replicas while using PVC

11/12/2020

I have an Open Source project that I deployed in Kubernetes. It has a master, data, and query pod. The data pod is used to store the data. Unfortunately, when the data pod goes down it loses all its stored data as well. So I figured, I have to use a StatefulSet with PVC. I started looking at PVC samples. Suppose I create a PV:

kind: PersistentVolume
metadata:
  name: task-pv-volume1
  labels:
    type: local
spec:
  storageClassName: local
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data1"

And then I have a Claim and a POD to bind it. This is the sample Stateful Set:

kind: StatefulSet
metadata:
  name: myweb
spec:
  replicas: 2
...
        volumeMounts:
        - name: data
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: local
      resources:
        requests:
          storage: 100Mi

Now when I run a sample job, I realize that only one of the 2 pods is active. The other pod fails saying:

Warning  FailedScheduling  <unknown>        0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.

which I believe is because I created only one PV. So I go and create a new PV,

path: "/mnt/data2"

and the pod is up and running.

In my prod setup, I want this to happen automatically. Which means I do not want to create these PVs for each replica. Is there a way to do this? Is this taken care of by Kube in some way?

Appreciate the help. If you have any further questions, please reply. I'll try my best to resolve it.

-- CSUNNY
kubernetes
kubernetes-pvc
statefulset

1 Answer

11/13/2020

Currently the Kubernetes local storage provisioner is static and not dynamic. A dynamic provisioner would allow automatic spinning up of Volumes as mentioned in the post.

As the default Kubernetes local storage provisioner does not support this you will unfortuantely have to find another storage provisioner.

I would suggest looking at Ranchers local-path-provisioner: https://github.com/rancher/local-path-provisioner

It advertises in its overview

Local Path Provisioner provides a way for the Kubernetes users to utilize the local storage in each node. Based on the user configuration, the Local Path Provisioner will create hostPath based persistent volume on the node automatically. It utilizes the features introduced by Kubernetes Local Persistent Volume feature, but make it a simpler solution than the built-in local volume feature in Kubernetes.

Compare to built-in Local Persistent Volume feature in Kubernetes

Pros

Dynamic provisioning the volume using hostPath. Currently the Kubernetes Local Volume provisioner cannot do dynamic provisioning for the local volumes.

Cons

No support for the volume capacity limit currently. The capacity limit will be ignored for now.

You can install it by running

kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml

and then changing your storageClassName in your stateful set to local-path

-- Justin Tamblyn
Source: StackOverflow