kubernetes : deployment fails Error response from daemon: OCI runtime create failed:

12/4/2018

I have set up a two node cluster on raspberry pi and am trying to set up a ghost blog platform. I created a deployment and I see that the containers are crashing with the status "CrashLoopback" ANy help with the troubleshooting is much appreciated. thanks

Warning  Failed     7m13s                 kubelet, kube-node-2  Error: failed to start container "ghost": Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/docker/volumes/994c880776e4f19087b08dbd0b36362f71ed32a412691f0aac28c6f024ea20ed/_data\\\" to rootfs \\\"/var/lib/docker/overlay2/c83d0b7a7a4e0f576c63c39907aad3d54b4657d6520ca47fbd201e1e3b3a9fe3/merged\\\" at \\\"/var/lib/docker/overlay2/c83d0b7a7a4e0f576c63c39907aad3d54b4657d6520ca47fbd201e1e3b3a9fe3/merged/var/lib/ghost/content\\\" caused \\\"mkdir /var/lib/docker/overlay2/c83d0b7a7a4e0f576c63c39907aad3d54b4657d6520ca47fbd201e1e3b3a9fe3/merged/var/lib/ghost/content: permission denied\\\"\"": unknown
  Normal   Pulling    6m31s (x5 over 10m)   kubelet, kube-node-2  pulling image "arm32v7/ghost"
  Normal   Created    6m5s (x5 over 9m49s)  kubelet, kube-node-2  Created container
  Normal   Pulled     6m5s (x5 over 9m50s)  kubelet, kube-node-2  Successfully pulled image "arm32v7/ghost"
  Warning  BackOff    18s (x29 over 8m5s)   kubelet, kube-node-2  Back-off restarting failed container

I see that it has something to down with permissions but unsure what these permissions are. Here are the output from PV and PVC

 kubectl describe pv pv-ghost
Name:            pv-ghost
Labels:          pv=pv-ghost
Annotations:     pv.kubernetes.io/bound-by-controller: yes
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:
Status:          Bound
Claim:           default/pvc-ghost
Reclaim Policy:  Retain
Access Modes:    RWX
Capacity:        3Gi
Node Affinity:   <none>
Message:
Source:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    192.168.178.50
    Path:      /mnt/ud-hdd/ghost/data
    ReadOnly:  false
Events:        <none>

kubectl describe pvc pvc-ghost
Name:          pvc-ghost
Namespace:     default
StorageClass:
Status:        Bound
Volume:        pv-ghost
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      3Gi
Access Modes:  RWX
Events:        <none>
Mounted By:    ghost-deployment-6cd67cbb7f-5vh5j
               ghost-deployment-6cd67cbb7f-67xtr
               ghost-deployment-6cd67cbb7f-fvq8w
               ghost-deployment-6cd67cbb7f-sgksk
               ghost-deployment-6cd67cbb7f-wgtzf
               ghost-deployment-6cd67cbb7f-xrrdc

Error after provide nfs server the root access

kubectl logs ghost-deployment-5759f4855c-294cl
tar: /var/lib/ghost/content.orig: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
-- IT_novice
kubernetes

1 Answer

12/4/2018

The permission denied error relates to the following operation: mkdir /var/lib/docker/overlay2/c83d0b7a7a4e0f576c63c39907aad3d54b4657d6520ca47fbd201e1e3b3a9fe3/merged/var/lib/ghost/content: permission denied

It's not clear from the info you've provided but if I had to guess you're trying to create a directory on the NFS share. If so, you most likely don't have write permission by the NFS server to write to that path.

-- Dimitri Vorobiev
Source: StackOverflow