K
Q

Debug failed kubernetes deployment

9/15/2019

I am experimenting with Kubernetes on Digital Ocean. As a testcase, i am trying to deploy a Jenkins instance to my cluster with a persistent volume.

My deployment yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins-deployment
  labels:
    app: jenkins
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      containers:
      - name: jenkins
        image: jenkins/jenkins:lts
        ports:
          - containerPort: 8080
        volumeMounts:
          - name: jenkins-home
            mountPath: /var/jenkins_home
      volumes:
        - name: jenkins-home
          persistentVolumeClaim:
            claimName: jenkins-pvc

My PV Claim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: do-block-storage
  resources:
    requests:
      storage: 30Gi

For some reason the pod keeps ending up in a CrashLoopBackOff state.

kubectl describe pod <podname> gives me

Name:           jenkins-deployment-bb5857d76-j2f2w
Namespace:      default
Priority:       0
Node:           cc-pool-bg6c/10.138.123.186
Start Time:     Sun, 15 Sep 2019 22:18:56 +0200
Labels:         app=jenkins
                pod-template-hash=bb5857d76
Annotations:    <none>
Status:         Running
IP:             10.244.0.166
Controlled By:  ReplicaSet/jenkins-deployment-bb5857d76
Containers:
  jenkins:
    Container ID:   docker://4eaadebb917001d8d3eaaa3b043e1b58b6269f929b9e95c4b08d88b0098d29d6
    Image:          jenkins/jenkins:lts
    Image ID:       docker-pullable://jenkins/jenkins@sha256:7cfe34701992434cc08bfd40e80e04ab406522214cf9bbefa57a5432a123b340
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sun, 15 Sep 2019 22:35:14 +0200
      Finished:     Sun, 15 Sep 2019 22:35:14 +0200
    Ready:          False
    Restart Count:  8
    Environment:    <none>
    Mounts:
      /var/jenkins_home from jenkins-home (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wd6p7 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  jenkins-home:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  jenkins-pvc
    ReadOnly:   false
  default-token-wd6p7:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-wd6p7
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age                  From                     Message
  ----     ------                  ----                 ----                     -------
  Normal   Scheduled               19m                  default-scheduler        Successfully assigned default/jenkins-deployment-bb5857d76-j2f2w to cc-pool-bg6c
  Normal   SuccessfulAttachVolume  19m                  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-cb772fdb-492b-4ef5-a63e-4e483b8798fd"
  Normal   Pulled                  17m (x5 over 19m)    kubelet, cc-pool-bg6c    Container image "jenkins/jenkins:lts" already present on machine
  Normal   Created                 17m (x5 over 19m)    kubelet, cc-pool-bg6c    Created container jenkins
  Normal   Started                 17m (x5 over 19m)    kubelet, cc-pool-bg6c    Started container jenkins
  Warning  BackOff                 4m8s (x72 over 19m)  kubelet, cc-pool-bg6c    Back-off restarting failed container

Could anyone help me point out what is wrong here, or where to look for that matter?

Many thanks in advance.

-- Guardian
jenkins
kubernetes

Similar Questions

KubeDB on minikube mount local directory
Unify service endpoints in kubernetes
How does Kubernetes Horizontal Pod Autoscaler calculate CPU Utilization for Multi Container Pods?
Is it possible to send metrics to prometheus?
EKS provisioned LoadBalancers reference all nodes in the cluster. If the pods reside on 1 or 2 nodes is this efficient as the ELB traffic increases?
kubernates rook+ceph OSD full delete data
Kubernetes loadbalance between monolith and microservices

2 Answers

9/15/2019

I cannot add comment (not enough reputation :D)

Looking at the pod logs may give some idea kubectl logs <podname>

PVC seems to be allocated and attached without issues. I had similar issues deploying nginx container a while ago, the issue was nginx container requires specific UID on host to work. Check if jenkins container needs specific UID.

Also, you may need to set required permissions on the volume (depending on jenkins requirement).

-- YAP
Source: StackOverflow
9/15/2019

Looks like you don't have permission to write to the volume. Try running the container as root using security contexts:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins-deployment
  labels:
    app: jenkins
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      securityContext:
        fsGroup: 1000 
        runAsUser: 0
      containers:
      - name: jenkins
        image: jenkins/jenkins:lts
        ports:
          - containerPort: 8080
        volumeMounts:
          - name: jenkins-home
            mountPath: /var/jenkins_home
      volumes:
        - name: jenkins-home
          persistentVolumeClaim:
            claimName: jenkins-pvc
-- Alassane Ndiaye
Source: StackOverflow