WSL Kubernetes Pod stuck in ContainerCreating state due to volume mount

5/13/2020

I am working with Docker Desktop on Windows 10 as well as Windows Subsystem for Linux (WSL). I have a containerized app that I deploy to the local K8s cluster (courtesy of Docker Desktop). Typical story: all was working fine until one day a Docker Desktop update came and ruined everything). DD version I have now is 2.3.0.2 stable.

I have a pod with MySQL with defined pv, pvc and storage class. When I deploy my app to the cluster I can see that pv and pvc are bound but the pod is stuck at ContainerCreating state:

$ kubectl describe pod mysql-6779d8fb8b-d25wz
Name:           mysql-6779d8fb8b-d25wz
Namespace:      typo3-connector
Priority:       0
Node:           docker-desktop/192.168.65.3
Start Time:     Wed, 13 May 2020 14:21:43 +0200
Labels:         app=mysql
                pod-template-hash=6779d8fb8b
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/mysql-6779d8fb8b
Containers:
  mysql:
    Container ID:
    Image:          lw-mysql
    Image ID:
    Port:           3306/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      mysql-credentials  Secret  Optional: false
    Environment:         <none>
    Mounts:
      /var/lib/mysql from mysql-persistent-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wr6g9 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  mysql-persistent-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mysql-pv-claim
    ReadOnly:   false
  default-token-wr6g9:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-wr6g9
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                From                     Message
  ----     ------            ----               ----                     -------
  Warning  FailedScheduling  <unknown>          default-scheduler        persistentvolumeclaim "mysql-pv-claim" not found
  Warning  FailedScheduling  <unknown>          default-scheduler        persistentvolumeclaim "mysql-pv-claim" not found
  Normal   Scheduled         <unknown>          default-scheduler        Successfully assigned typo3-connector/mysql-6779d8fb8b-d25wz to docker-desktop
  Warning  FailedMount       35s (x8 over 99s)  kubelet, docker-desktop  MountVolume.NewMounter initialization failed for volume "mysql-pv" : path "/c/kubernetes/typo3-8/mysql-storage/" does not exist

The error is

MountVolume.NewMounter initialization failed for volume "mysql-pv" : path "/c/kubernetes/typo3-8/mysql-storage/" does not exist

But the path actually exists on disk (in WSL that is).

The pv:

$ kubectl describe pv mysql-pv
Name:              mysql-pv
Labels:            <none>
Annotations:       kubectl.kubernetes.io/last-applied-configuration:
                     {"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"mysql-pv"},"spec":{"accessModes":["ReadWriteOnce"],"capa...
                   pv.kubernetes.io/bound-by-controller: yes
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      local-storage
Status:            Bound
Claim:             typo3-connector/mysql-pv-claim
Reclaim Policy:    Retain
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          2Gi
Node Affinity:
  Required Terms:
    Term 0:        kubernetes.io/hostname in [docker-desktop]
Message:
Source:
    Type:  LocalVolume (a persistent volume backed by local storage on a node)
    Path:  /c/kubernetes/typo3-8/mysql-storage/
Events:    <none>

The pvc:

$ kubectl describe pvc mysql-pv-claim
Name:          mysql-pv-claim
Namespace:     typo3-connector
StorageClass:  local-storage
Status:        Bound
Volume:        mysql-pv
Labels:        <none>
Annotations:   kubectl.kubernetes.io/last-applied-configuration:
                 {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"mysql-pv-claim","namespace":"typo3-connector"},"spe...
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      2Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Mounted By:    mysql-6779d8fb8b-d25wz
Events:        <none>

I tried running it from PowerShell but no luck. I get the same result. But it was all working fine before the update.

Is it configuration based issue?

EDIT Here's the manifest file:

apiVersion: v1
kind: Namespace
metadata:
  name: typo3-connector
---
apiVersion: v1
data:
  MYSQL_PASSWORD: ZHVtbXk=
  MYSQL_ROOT_PASSWORD: ZHVtbXk=
  MYSQL_USER: ZHVtbXk=
kind: Secret
metadata:
  name: mysql-credentials
  namespace: typo3-connector
type: Opaque
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: mysql
  name: mysql
  namespace: typo3-connector
spec:
  ports:
  - name: mysql-backend
    port: 3306
    protocol: TCP
  selector:
    app: mysql
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  namespace: typo3-connector
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - envFrom:
        - secretRef:
            name: mysql-credentials
        image: mysql:5.6
        imagePullPolicy: IfNotPresent
        name: mysql
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - mountPath: /var/lib/mysql
          name: mysql-persistent-storage
      imagePullSecrets:
      - name: lwdockerregistry
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 2Gi
  local:
    path: /c/kubernetes/typo3-8/mysql-storage/
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - docker-desktop
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
  namespace: typo3-connector
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: local-storage
-- Ventus
docker
kubernetes
persistent-volumes
windows-subsystem-for-linux

1 Answer

5/15/2020

As described in the following example taken from here, the path shouldn't end with /.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-local-pv
spec:
  capacity:
    storage: 500Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /mnt/disks/vol1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - my-node

Changing your path to path: /c/kubernetes/typo3-8/mysql-storage makes the magic and your PVC works as designed.

-- mWatney
Source: StackOverflow