Can't Share a Persistent Volume Claim for an EBS Volume between Apps

7/3/2017

Is it possible to share a single persistent volume claim (PVC) between two apps (each using a pod)?

I read: Share persistent volume claims amongst containers in Kubernetes/OpenShift but didn't quite get the answer.

I tried to added a PHP app, and MySQL app (with persistent storage) within the same project. Deleted the original persistent volume (PV) and created a new one with read,write,many mode. I set the root password of the MySQL database, and the database works.

Then, I add storage to the PHP app using the same persistent volume claim with a different subpath. I found that I can't turn on both apps. After I turn one on, when I try to turn on the next one, it get stuck at creating container.

MySQL .yaml of the deployment step at openshift:

  ...
  template:
    metadata:
      creationTimestamp: null
      labels:
        name: mysql
    spec:
      volumes:
        - name: mysql-data
          persistentVolumeClaim:
            claimName: mysql
      containers:
        - name: mysql
        ...
          volumeMounts:
            - name: mysql-data
              mountPath: /var/lib/mysql/data
              subPath: mysql/data
          ...
          terminationMessagePath: /dev/termination-log
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst

PHP .yaml from deployment step:

 template:
    metadata:
      creationTimestamp: null
      labels:
        app: wiki2
        deploymentconfig: wiki2
    spec:
      volumes:
        - name: volume-959bo  <<----
          persistentVolumeClaim:
            claimName: mysql
      containers:
        - name: wiki2
          ...
          volumeMounts:
            - name: volume-959bo
              mountPath: /opt/app-root/src/w/images
              subPath: wiki/images
          terminationMessagePath: /dev/termination-log
          imagePullPolicy: Always
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext: {}

The volume mount names are different. But that shouldn't make the two pods can't share the PVC. Or, the problem is that they can't both mount the same volume at the same time?? I can't get the termination log at /dev because if it can't mount the volume, the pod doesn't start, and I can't get the log.

The PVC's .yaml (oc get pvc -o yaml)

apiVersion: v1
items:
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-class: ebs
      volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
    creationTimestamp: YYYY-MM-DDTHH:MM:SSZ
    name: mysql
    namespace: abcdefghi
    resourceVersion: "123456789"
    selfLink: /api/v1/namespaces/abcdefghi/persistentvolumeclaims/mysql
    uid: ________-____-____-____-____________

  spec:
    accessModes:
    - ReadWriteMany
    resources:
      requests:
        storage: 1Gi
    volumeName: pvc-________-____-____-____-____________
  status:
    accessModes:
    - ReadWriteMany
    capacity:
      storage: 1Gi
    phase: Bound
kind: List
metadata: {}
resourceVersion: ""
selfLink: ""

Suspicious Entries from oc get events

Warning    FailedMount   {controller-manager }   
    Failed to attach volume "pvc-________-____-____-____-____________" 
    on node "ip-172-__-__-___.xx-xxxx-x.compute.internal" 
with: 
    Error attaching EBS volume "vol-000a00a00000000a0" to instance 
    "i-1111b1b11b1111111": VolumeInUse: vol-000a00a00000000a0 is 
    already attached to an instance

Warning   FailedMount   {kubelet ip-172-__-__-___.xx-xxxx-x.compute.internal}   
    Unable to mount volumes for pod "the pod for php app": 
    timeout expired waiting for volumes to attach/mount for pod "the pod". 
    list of unattached/unmounted volumes=
        [volume-959bo default-token-xxxxx]

I tried to:

  1. turn on the MySQL app first, and then try to turn on the PHP app
  2. found php app can't start
  3. turn off both apps
  4. turn on the PHP app first, and then try to turn on the MySQL app.
  5. found mysql app can't start

The strange thing is that the event log never says it can't mount volume for the MySQL app.

The remaining volumen to mount is either default-token-xxxxx, or volume-959bo (the volume name in PHP app), but never mysql-data (the volume name in MySQL app).

-- hamster on wheels
kubernetes
openshift

1 Answer

7/3/2017

So the error seems to be caused by the underlying storage you are using, in this case EBS. The OpenShift docs actually specifically state that this is the case for block storage, see here.

I know this will work for both NFS and Glusterfs storage, and have done this in numerous projects using these storage type but unfortunately, in your case it's not supported

-- PhilipGough
Source: StackOverflow