No way to apply change to Deployment binded with ReadWriteOnce PV in gCloud Kubernetes engine?

10/22/2018

As the GCE Disk does not support ReadWriteMany , I have no way to apply change to Deployment but being stucked at ContainerCreating with FailedAttachVolume .

So here's my setting:

1. PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
  labels:
    app: mysql
spec:
  storageClassName: "standard"
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

2. Service

apiVersion: v1
kind: Service
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  type: ClusterIP
  ports:
    - protocol: TCP
      port: 3306
      targetPort: 3306
  selector:
    app: mysql

3. Deployment

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - image: mysql/mysql-server
        name: mysql
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /mysql-data
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim

Which these are all fine for creating the PVC, svc and deployment. Pod and container started successfully and worked as expected.


However when I tried to apply change by: kubectl apply -f mysql_deployment.yaml

Firstly, the pods were sutcked with the existing one did not terminate and the new one would be creating forever.

NAME             READY     STATUS              RESTARTS   AGE
mysql-nowhash    1/1       Running             0          2d
mysql-newhash    0/2       ContainerCreating   0          15m

Secondly from the gCloud console, inside the pod that was trying to create, I got two crucial error logs:

1 of 2 FailedAttachVolume

Multi-Attach error for volume "pvc-<hash>" Volume is already exclusively attached to one node and can't be attached to another  FailedAttachVolume

2 of 2 FailedMount

Unable to mount volumes for pod "<pod name and hash>": timeout expired waiting for volumes to attach/mount for pod "default"/"<pod name and hash>". list of unattached/unmounted volumes=[mysql-persistent-storage] 

What I could immediately think of is the ReadWriteOnce capability of gCloud PV. Coz the kubernetes engine would create a new pod before terminating the existing one. So under ReadWriteOnce it can never create a new pod and claim the existing pvc...

Any idea or should I use some other way to perform deployment updates? appreciate for any contribution and suggestion =)

remark: my current work-around is to create an interim NFS pod to make it like a ReadWriteMany pvc, this worked but sounds stupid... requiring an additional storage i/o overhead to facilitate deployment update ?.. =P

-- Michael Linus
gcloud
google-cloud-platform
kubernetes
persistent-volumes

1 Answer

10/22/2018

The reason is that if you are applying UpdateStrategy: RollingUpdate (as it is default) k8s waits for the new Container to become ready before shutting down the old one. You can change this behaviour by applying UpdateStrategy: Recreate

https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#strategy

-- Louis Baumann
Source: StackOverflow