Kubernetes PVC with AccessMode: ReadWriteOnce

5/25/2017

Our scenarios:

We use ceph rbd to store some machine learning training dataset, the workflow as below:

Create a ceph-rbd pvc pvc-tranining-data with AccessMode: ReadWriteOnce.
Create a write job with 1 pod to mount pvc-training-data and write training data in to pvc-training-data.
After writing the training data into pvc-training-data, the container will exit and the pvc-trainiing-data pvc is unmounted by k8s, the write job is done.
Create a read job with n pods where n >= 1 to mount pvc-training-data with readOnly: true to consume the training data. btw: we use k8s 1.6.1

So far the workflow works well for our use scenarios, but I have some questions about PVC AccessMode and ceph rbd with AccessMode: ReadWriteOnce.

  1. How to understand AccessModes: ReadOnlyMany, ReadWriteOnce, ReadWriteMany? I think the usage scope is ReadOnlyMany < ReadWriteOnce < ReadWriteMay, so if I apply a PVC with AccessMode: ReadWriteOnce, it is ok I use it as AccessMode: ReadOnlyMany PVC, am I right?

  2. ceph rbd is a block device, each container(on different host) mount the same ceph rbd device will have their own filesystem, so the only allowed AccessMode is ReadOnlyMany or ReadWriteOnce, should we impose restrictions on ReadWriteOnce usage in k8s code?

  1. If a ReadWriteOnce pvc is mounted by a Pod with readOnly:false then no more Pod can mount this device, until it is unmounted.
  2. If a ReadWriteOnce pvc is mounted by a Pod with readOnly:true, it only can be mounted to other Pod as long as they set readOnly: true.
  3. there's no restrictions on container with in the same Pod, as they share the same filesystem from host ${KUBELET_ROOT}/plugins/{xx}/
-- Wei Wei
kubernetes

2 Answers

5/25/2017

Regarding your first question:

  1. How to understand AccessModes: ReadOnlyMany, ReadWriteOnce, ReadWriteMany? I think the usage scope is ReadOnlyMany < ReadWriteOnce < ReadWriteMay, so if I apply a PVC with AccessMode: ReadWriteOnce, it is ok I use it as AccessMode: ReadOnlyMany PVC, am I right?

The docs clearly state:

Important! A volume can only be mounted using one access mode at a
time, even if it supports many.

Your second question was not clear to me. But I think it may not be valid considering the answer to the first one?

-- Oswin Noetzelmann
Source: StackOverflow

10/18/2019

To answer this question, it might be useful to take a look at source code. As I have faced with an issue in our cluster migration, here is the case: We have a cluster with v1.9 and we have a PV which has accessMode: ReadWriteMany (RWX) and PVC with accessMode: ReadWriteOnce and those two can bound without any error.

We migrate some of our applications to new cluster(v1.12) and in new cluster PVC binding gives an error:

Cannot bind to requested volume "volume-name": incompatible accessMode

I searched the error in the source code of v1.12 and I saw those lines:

v1.12:

//checkVolumeSatisfyClaim checks if the volume requested by the claim satisfies the requirements of the claim
func checkVolumeSatisfyClaim(volume *v1.PersistentVolume, claim *v1.PersistentVolumeClaim) error {
    requestedQty := claim.Spec.Resources.Requests[v1.ResourceName(v1.ResourceStorage)]
    requestedSize := requestedQty.Value()

    // check if PV's DeletionTimeStamp is set, if so, return error.
    if utilfeature.DefaultFeatureGate.Enabled(features.StorageObjectInUseProtection) {
        if volume.ObjectMeta.DeletionTimestamp != nil {
            return fmt.Errorf("the volume is marked for deletion")
        }
    }

    volumeQty := volume.Spec.Capacity[v1.ResourceStorage]
    volumeSize := volumeQty.Value()
    if volumeSize < requestedSize {
        return fmt.Errorf("requested PV is too small")
    }

    requestedClass := v1helper.GetPersistentVolumeClaimClass(claim)
    if v1helper.GetPersistentVolumeClass(volume) != requestedClass {
        return fmt.Errorf("storageClassName does not match")
    }

    isMisMatch, err := checkVolumeModeMisMatches(&claim.Spec, &volume.Spec)
    if err != nil {
        return fmt.Errorf("error checking volumeMode: %v", err)
    }
    if isMisMatch {
        return fmt.Errorf("incompatible volumeMode")
    }

    if !checkAccessModes(claim, volume) {
        return fmt.Errorf("incompatible accessMode")
    }

    return nil
}

// Returns true if PV satisfies all the PVC's requested AccessModes
func checkAccessModes(claim *v1.PersistentVolumeClaim, volume *v1.PersistentVolume) bool {
    pvModesMap := map[v1.PersistentVolumeAccessMode]bool{}
    for _, mode := range volume.Spec.AccessModes {
        pvModesMap[mode] = true
    }

    for _, mode := range claim.Spec.AccessModes {
        _, ok := pvModesMap[mode]
        if !ok {
            return false
        }
    }
    return true
}

v1.9:

//checkVolumeSatisfyClaim checks if the volume requested by the claim satisfies the requirements of the claim
func checkVolumeSatisfyClaim(volume *v1.PersistentVolume, claim *v1.PersistentVolumeClaim) error {
    requestedQty := claim.Spec.Resources.Requests[v1.ResourceName(v1.ResourceStorage)]
    requestedSize := requestedQty.Value()
    isMisMatch, err := checkVolumeModeMisMatches(&claim.Spec, &volume.Spec)
    if err != nil {
        return fmt.Errorf("error checking if volumeMode was a mismatch: %v", err)
    }

    volumeQty := volume.Spec.Capacity[v1.ResourceStorage]
    volumeSize := volumeQty.Value()
    if volumeSize < requestedSize {
        return fmt.Errorf("Storage capacity of volume[%s] requested by claim[%v] is not enough", volume.Name, claimToClaimKey(claim))
    }

    requestedClass := v1helper.GetPersistentVolumeClaimClass(claim)
    if v1helper.GetPersistentVolumeClass(volume) != requestedClass {
        return fmt.Errorf("Class of volume[%s] is not the same as claim[%v]", volume.Name, claimToClaimKey(claim))
    }

    if isMisMatch {
        return fmt.Errorf("VolumeMode[%v] of volume[%s] is incompatible with VolumeMode[%v] of claim[%v]", volume.Spec.VolumeMode, volume.Name, claim.Spec.VolumeMode, claim.Name)
    }

    return nil
}

// checkVolumeModeMatches is a convenience method that checks volumeMode for PersistentVolume
// and PersistentVolumeClaims along with making sure that the Alpha feature gate BlockVolume is
// enabled.
// This is Alpha and could change in the future.
func checkVolumeModeMisMatches(pvcSpec *v1.PersistentVolumeClaimSpec, pvSpec *v1.PersistentVolumeSpec) (bool, error) {
    if utilfeature.DefaultFeatureGate.Enabled(features.BlockVolume) {
        if pvSpec.VolumeMode != nil && pvcSpec.VolumeMode != nil {
            requestedVolumeMode := *pvcSpec.VolumeMode
            pvVolumeMode := *pvSpec.VolumeMode
            return requestedVolumeMode != pvVolumeMode, nil
        } else {
            // This also should retrun an error, this means that
            // the defaulting has failed.
            return true, fmt.Errorf("api defaulting for volumeMode failed")
        }
    } else {
        // feature gate is disabled
        return false, nil
    }
}

When you take a look at v1.12 code in checkAccessModes function, it puts volume accessModes into a Map and searchs PVC accessModes in that map, if it can not find PVC accessMode in that map, it returns false which is causes the incompatible accessMode error.

So why we don't get this error in v1.9 ? Because it has a different control in checkVolumeModeMisMatches function. It checks an alpha feature which is false by default called BlockVolume. Because of it is false, it does not encounter into

if isMisMatch {
    return fmt.Errorf("VolumeMode[%v] of volume[%s] is incompatible with VolumeMode[%v] of claim[%v]", volume.Spec.VolumeMode, volume.Name, claim.Spec.VolumeMode, claim.Name)
}

code block.

you can check the BlockVolume feature on master node with :

ps aux | grep apiserver | grep feature-gates

I hope it clarifies your question. Especially the checkAccessModes function in v1.12 also in the master branch at the moment does the PV-PVC accessMode control.

-- Emre Savcı
Source: StackOverflow