K
Q

Why am I getting a CrashLoopBackOff on this kubernetes example?

4/11/2019

I am following the instruction to create a simple stateful app with mysql following the instructions from kubernetes official documentation, but it does work for me and I wonder if any of you can test it in his own GCP in two minutes and see if I am the only one having problems of the example just does not work:

https://kubernetes.io/docs/tasks/run-application/run-single-instance-stateful-application/

These are the files from the documentation:

application/mysql/mysql-deployment.yaml

apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  ports:
  - port: 3306
  selector:
    app: mysql
  clusterIP: None
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - image: mysql:5.6
        name: mysql
        env:
          # Use secret in real usage
        - name: MYSQL_ROOT_PASSWORD
          value: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim

application/mysql/mysql-pv.yaml

kind: PersistentVolume
apiVersion: v1
metadata:
  name: mysql-pv-volume
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

NOTE: That I am doing several attempts so the pod names might differ...

It looks like everything went fine, but I get a CrashLoopBackOff:

xxx@cloudshell:~ (academic-veld-230622)$ gcloud container clusters get-credentials standard-cluster-1 --zone us-central1-a --project academic-veld-230622
Fetching cluster endpoint and auth data.
kubeconfig entry generated for standard-cluster-1.
xx@cloudshell:~ (academic-veld-230622)$ kubectl apply -f https://k8s.io/examples/application/mysql/mysql-pv.yaml
persistentvolume/mysql-pv-volume created
persistentvolumeclaim/mysql-pv-claim created
xxx@cloudshell:~ (academic-veld-230622)$ kubectl apply -f https://k8s.io/examples/application/mysql/mysql-deployment.yaml
service/mysql created
deployment.apps/mysql created
@cloudshell:~ (academic-veld-230622)$ kubectl describe deployment mysql
Name:               mysql
Namespace:          default
CreationTimestamp:  Thu, 11 Apr 2019 18:46:58 +0200
Labels:             <none>
Annotations:        deployment.kubernetes.io/revision=1
                    kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"mysql","namespace":"default"},"spec":{"selector":{"matchLabels":{"app"...
Selector:           app=mysql
Replicas:           1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType:       Recreate
MinReadySeconds:    0
Pod Template:
  Labels:  app=mysql
  Containers:
   mysql:
    Image:      mysql:5.6
    Port:       3306/TCP
    Host Port:  0/TCP
    Environment:
      MYSQL_ROOT_PASSWORD:  password
    Mounts:
      /var/lib/mysql from mysql-persistent-storage (rw)
  Volumes:
   mysql-persistent-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mysql-pv-claim
    ReadOnly:   false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      False   MinimumReplicasUnavailable
  Progressing    True    ReplicaSetUpdated
OldReplicaSets:  <none>
NewReplicaSet:   mysql-fb75876c6 (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  34s   deployment-controller  Scaled up replica set mysql-fb75876c6 to 1
xxxx@cloudshell:~ (academic-veld-230622)$ kubectl get pods -l app=mysql
NAME                    READY     STATUS              RESTARTS   AGE
mysql-fb75876c6-522j9   0/1       RunContainerError   4          1m

xxx@cloudshell:~ (academic-veld-230622)$ kubectl get pods -l app=mysql
NAME                    READY     STATUS             RESTARTS   AGE
mysql-fb75876c6-522j9   0/1       CrashLoopBackOff   6          7m

@cloudshell:~ (academic-veld-230622)$ kubectl describe pvc mysql-pv-claim
Name:          mysql-pv-claim
Namespace:     default
StorageClass:  manual
Status:        Bound
Volume:        mysql-pv-volume
Labels:        <none>
Annotations:   kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"mysql-pv-claim","namespace":"default"},"spec":{"accessModes":["R...
               pv.kubernetes.io/bind-completed=yes
               pv.kubernetes.io/bound-by-controller=yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      20Gi
Access Modes:  RWO
Events:        <none>

Logs on the second attempt in a bigger machine:

@cloudshell:~ (academic-veld-230622)$ kubectl logs mysql-fb75876c6-ctchn --previous
failed to open log file "/var/log/pods/68c34d6f-5c7d-11e9-9029-42010a800043/mysql/5.log": open /var/log/pods/68c34d6f-5c7d-11e9-9029-42010a800043/mysql/5.log: no such file or directorymasuareza@cloudshell:~ (academic-veld-230622)$

Going into the console:

https://console.cloud.google.com/logs/viewer?interval=NO_LIMIT&project=academic-veld-230622&authuser=0&minLogLevel=0&expandAll=false×tamp=2019-04-11T17%3A25%3A37.805000000Z&customFacets&limitCustomFacetWidth=true&advancedFilter=resource.type%3D%22k8s_cluster%22%0Aresource.labels.project_id%3D%22academic-veld-230622%22%0Aresource.labels.cluster_name%3D%22standard-cluster-1%22%0Aresource.labels.location%3D%22us-central1-a%22%0Atimestamp%3D%222019-04-11T17%3A15%3A30.999350000Z%22%0AinsertId%3D%22d99409fb-66f8-4eab-8066-6c8b976aaec8%22&scrollTimestamp=2019-04-11T17%3A15%3A30.999350000Z

Third attempt:

Showing logs from all time (CEST)

No older entries found matching current filter.  
2019-04-11 19:39:54.278 CEST
k8s.io
create
default:mysql-fb75876c6-r42x6:mysql-fb75876c6-r42x6
system:kube-scheduler
{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","authenticationInfo":{"principalEmail":"system:kube-scheduler"},"authorizationInfo":[{"granted":true,"permission":"io.k8s.core.v1.pods.binding.create","resource":"core/v1/namespaces/default/pods/mysql-fb75876c6-r42x6/binding/mysql-fb75876c6-…
Expand all | Collapse all {
 insertId:  "39605b90-16dd-470b-996f-b072fb262595"  
 labels: {…}  
 logName:  "projects/academic-veld-230622/logs/cloudaudit.googleapis.com%2Factivity"  
 operation: {…}  
 protoPayload: {…}  
 receiveTimestamp:  "2019-04-11T17:40:22.506801591Z"  
 resource: {…}  
 timestamp:  "2019-04-11T17:39:54.278100Z"  
}

DESCRIBE POD:

@cloudshell:~ (academic-veld-230622)$ kubectl describe pod mysql-fb75876c6-r42x6
Name:               mysql-fb75876c6-r42x6
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               gke-standard-cluster-1-default-pool-119c7a9c-5jp1/10.150.0.15
Start Time:         Thu, 11 Apr 2019 19:39:54 +0200
Labels:             app=mysql
                    pod-template-hash=963143272
Annotations:        kubernetes.io/limit-ranger=LimitRanger plugin set: cpu request for container mysql
Status:             Running
IP:                 10.48.0.13
Controlled By:      ReplicaSet/mysql-fb75876c6
Containers:
  mysql:
    Container ID:   docker://63fbbebe5d246f56299b0194ed34ca3614349db1ab96251e23d098b0efbcac4b
    Image:          mysql:5.6
    Image ID:       docker-pullable://mysql@sha256:5ab881bc5abe2ac734d9fb53d76d984cc04031159152ab42edcabbd377cc0859
    Port:           3306/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       ContainerCannotRun
      Message:      error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system
      Exit Code:    128
      Started:      Thu, 11 Apr 2019 19:45:40 +0200
      Finished:     Thu, 11 Apr 2019 19:45:40 +0200
    Ready:          False
    Restart Count:  6
    Requests:
      cpu:  100m
    Environment:
      MYSQL_ROOT_PASSWORD:  password
    Mounts:
      /var/lib/mysql from mysql-persistent-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-rrhql (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  mysql-persistent-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mysql-pv-claim
    ReadOnly:   false
  default-token-rrhql:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-rrhql
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                 From                                                        Message
  ----     ------     ----                ----                                                        -------
  Normal   Scheduled  10m                 default-scheduler                                           Successfully assigned default/mysql-fb75876c6-r42x6 to gke-standard-cluster-1-default-pool-119c7a9c-5jp1
  Normal   Pulling    10m                 kubelet, gke-standard-cluster-1-default-pool-119c7a9c-5jp1  pulling image "mysql:5.6"
  Normal   Pulled     10m                 kubelet, gke-standard-cluster-1-default-pool-119c7a9c-5jp1  Successfully pulled image "mysql:5.6"
  Normal   Created    9m (x5 over 10m)    kubelet, gke-standard-cluster-1-default-pool-119c7a9c-5jp1  Created container
  Warning  Failed     9m (x5 over 10m)    kubelet, gke-standard-cluster-1-default-pool-119c7a9c-5jp1  Error: failed to start container "mysql": Error response from daemon: error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system
  Normal   Pulled     9m (x4 over 10m)    kubelet, gke-standard-cluster-1-default-pool-119c7a9c-5jp1  Container image "mysql:5.6" already present on machine
  Warning  BackOff    46s (x43 over 10m)  kubelet, gke-standard-cluster-1-default-pool-119c7a9c-5jp1  Back-off restarting failed container


failed to start container "6562e2c146ecf2087d438141550e385a1abf83de8ef1dd7a6fdca61d97576741": Error response from daemon: error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system: RunContainerError

PV & PVC:

(unique-poetry-233821)$ kubectl get pv
NAME              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                    STORAGECLASS   REASON    AGE
mysql-pv-volume   20Gi       RWO            Retain           Bound     default/mysql-pv-claim   manual                   30m

@cloudshell:~ (unique-poetry-233821)$ kubectl get pvc
NAME             STATUS    VOLUME            CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mysql-pv-claim   Bound     mysql-pv-volume   20Gi       RWO            manual         31m
-- Mike
kubernetes
mysql

Similar Questions

Does Spark Kubernetes suport the --py-files argument?
What is the correct way to set annotations on a Kubernetes Service in YAML
How to run pod with existing statefulset in kubernetes cluster?
continous health check of first server inside second server using nodejs
k8s ingress self-signed certificate won&#39;t be accepted

2 Answers

4/25/2019

Remove your PV definition. Create a StorageClass and it should work

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: manual
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-standard
  replication-type: none
-- Jorge M. Nures
Source: StackOverflow
5/21/2020
  • List the Persistent volume and persistent volume claim
kubectl get pvc,vc
  • Delete them in the following sequence
kubectl delete pvc <name>-pv-claim
kubectl delete pv <name>-pv
-- Rub21
Source: StackOverflow