kubernetes go-client PersistentVolumeClaim not provisioned on request, stuck in Pending state

3/27/2019

While using the go-client API after I use the api.PersistentVolumeClaims(namespace).Create(createOpts) call the PersistentVolumeClaim appears as a resource but stays in the Pending state. I do not see any events when using kubectl describe pvc, I also don't see any Volumes being created etc.

$ kubectl describe pvc --namespace=test -R
Name:          93007732-9d8c-406e-be99-f48faed3a061
Namespace:     test
StorageClass:  microk8s-hostpath
Status:        Pending
Volume:        93007732-9d8c-406e-be99-f48faed3a061
Labels:        <none>
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      0
Access Modes:  
VolumeMode:    Filesystem
Events:        <none>
Mounted By:    <none>

The code I am using is as follows:

        volume, errGo := uuid.NewRandom()                                                                                                                                                 
        if errGo != nil {                                                                                                                                                                 
                job.failed = kv.Wrap(errGo).With("stack", stack.Trace().TrimRuntime())                                                                                                    
                return job.failed                                                                                                                                                         
        }                                                                                                                                                                                 
        job.volume = volume.String()

        fs := v1.PersistentVolumeFilesystem
        createOpts := &v1.PersistentVolumeClaim{
                ObjectMeta: metav1.ObjectMeta{
                        Name:      job.volume,
                        Namespace: job.namespace,
                        UID:       types.UID(job.volume),
                },
                Spec: v1.PersistentVolumeClaimSpec{
                        AccessModes: []v1.PersistentVolumeAccessMode{v1.ReadWriteOnce},
                        Resources: v1.ResourceRequirements{
                                Requests: v1.ResourceList{
                                        v1.ResourceName(v1.ResourceStorage): resource.MustParse("10Gi"),
                                },
                        },
                        VolumeName: job.volume,
                        VolumeMode: &fs,
                },
                Status: v1.PersistentVolumeClaimStatus{
                        Phase:       v1.ClaimBound,
                        AccessModes: []v1.PersistentVolumeAccessMode{v1.ReadWriteOnce},
                        Capacity: v1.ResourceList{
                                v1.ResourceName(v1.ResourceStorage): resource.MustParse("10Gi"),
                        },
                },
        }

        api := Client().CoreV1()
        if _, errGo = api.PersistentVolumeClaims(namespace).Create(createOpts); errGo != nil {
                job.failed = kv.Wrap(errGo).With("stack", stack.Trace().TrimRuntime())
                return job.failed
        }

I have tried to find good examples for using the Create API with persistent volumes but most examples appear to be for watchers etc and so I spent quite sometime trying to reverse engineer code leading me explicitly setting the Status but this appears to have had zero impact. I also tried defaulting the VolumeMode within the Spec which did not help.

The examples I have read come from:

https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/persistentvolume/framework_test.go
https://godoc.org/k8s.io/api/core/v1#PersistentVolumeSpec
https://github.com/vladimirvivien/k8s-client-examples/tree/master/go/pvcwatch
https://medium.com/programming-kubernetes/building-stuff-with-the-kubernetes-api-part-4-using-go-b1d0e3c1c899

Does anyone know of actually example code for these APIs that goes beyond unit testing within the _test.go files, or can anyone provide any hints as to how to get the creation process actually rolling within the cluster ? I have assumed that the downstream resources needed for example the Volume etc are automatically provisioned when I attempt to create the Claim resource.

Many Thanks for taking a look if you got this far...

-- Karl Mutch
kubernetes
kubernetes-go-client
persistent-volume-claims

1 Answer

3/27/2019

What you are doing in the code looks correct. However, it looks like that your PVC can't find a matching PV to bind together.

It looks like you are using a hostPath PV (with a storage class) that doesn't support dynamic provisioning. Also, documented here.

So most likely you will have to create a hostPath PV so that your PVC can bind to it. The volume has to be equal or greater in size as what you are requesting in your PVC.

Another option is to use a Local volume that supports dynamic provisioning which is different from hostPath.

You can debug the dynamic provisioning and binding of the PVC/PV by looking at the kube-controller-manager logs on your K8s control plane leader.

-- Rico
Source: StackOverflow