helm install stable/jenkins failed: PodInitializing

12/10/2018

Version of Helm and Kubernetes: helm version: 2.12 kubernetes: 1.10.11

Which chart: stable/jenkins

What happened: Jenkins pod gets Init:CrashLoopBackOff.

From the descript pod section, it says /var/jenkins_config/apply_config.sh failed with error code 1.

How to reproduce it (as minimally and precisely as possible):

helm install --name jenkins -f \
helm-values/jenkins.yaml stable/jenkins \
--namespace kube-system

jenkins.yaml

# Default values for jenkins.
# This is a YAML-formatted file.
# Declare name/value pairs to be passed into your templates.
# name: value

## Overrides for generated resource names
# See templates/_helpers.tpl
# nameOverride:
# fullnameOverride:

Master:
  Name: jenkins-master
  Image: "jenkins/jenkins"
  ImageTag: "jdk11"
  ImagePullPolicy: "Always"
# ImagePullSecret: jenkins
  Component: "jenkins-master"
  UseSecurity: true
  # SecurityRealm:
  # Optionally configure a different AuthorizationStrategy using Jenkins XML
  # AuthorizationStrategy: |-
  #    <authorizationStrategy class="hudson.security.FullControlOnceLoggedInAuthorizationStrategy">
  #      <denyAnonymousReadAccess>true</denyAnonymousReadAccess>
  #    </authorizationStrategy>
  HostNetworking: false
  AdminUser: admin
  # AdminPassword: <defaults to random>
  resources:
    requests:
      cpu: "500m"
      memory: "512Mi"
    limits:
      cpu: "2000m"
      memory: "2048Mi"
  # Environment variables that get added to the init container (useful for e.g. http_proxy)
  # InitContainerEnv:
  #   - name: http_proxy
  #     value: "http://192.168.64.1:3128"
  # ContainerEnv:
  #   - name: http_proxy
  #     value: "http://192.168.64.1:3128"
  # Set min/max heap here if needed with:
  JavaOpts: "-Xms512m -Xmx1024m"
  # JenkinsOpts: ""
  # JenkinsUrl: ""
  # If you set this prefix and use ingress controller then you might want to set the ingress path below
  # JenkinsUriPrefix: "/jenkins"
  # Enable pod security context (must be `true` if RunAsUser or FsGroup are set)
  UsePodSecurityContext: true
  # Set RunAsUser to 1000 to let Jenkins run as non-root user 'jenkins' which exists in 'jenkins/jenkins' docker image.
  # When setting RunAsUser to a different value than 0 also set FsGroup to the same value:
  RunAsUser: 1000
  FsGroup: 1000
  ServicePort: 8080
  # For minikube, set this to NodePort, elsewhere use LoadBalancer
  # Use ClusterIP if your setup includes ingress controller
  ServiceType: ClusterIP
  # Master Service annotations
  ServiceAnnotations: {}
  # Master Service Labels
  ServiceLabels: {}
  #   service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
  # Used to create Ingress record (should used with ServiceType: ClusterIP)
  HostName: jenkins.aws.tapfeeds.com
  # NodePort: <to set explicitly, choose port between 30000-32767
  # Enable Kubernetes Liveness and Readiness Probes
  # ~ 2 minutes to allow Jenkins to restart when upgrading plugins. Set ReadinessTimeout to be shorter than LivenessTimeout.
  HealthProbes: true
  HealthProbesLivenessTimeout: 90
  HealthProbesReadinessTimeout: 60
  HealthProbeLivenessFailureThreshold: 12
  SlaveListenerPort: 50000
  DisabledAgentProtocols:
    - JNLP-connect
    - JNLP2-connect
  CSRF:
    DefaultCrumbIssuer:
      Enabled: true
      ProxyCompatability: true
  CLI: false
  # Kubernetes service type for the JNLP slave service
  # SETTING THIS TO "LoadBalancer" IS A HUGE SECURITY RISK: https://github.com/kubernetes/charts/issues/1341
  SlaveListenerServiceType: ClusterIP
  SlaveListenerServiceAnnotations: {}
  LoadBalancerSourceRanges:
  - 0.0.0.0/0
  # Optionally assign a known public LB IP
  # LoadBalancerIP: 1.2.3.4
  # Optionally configure a JMX port
  # requires additional JavaOpts, ie
  # JavaOpts: >
  #   -Dcom.sun.management.jmxremote.port=4000
  #   -Dcom.sun.management.jmxremote.authenticate=false
  #   -Dcom.sun.management.jmxremote.ssl=false
  # JMXPort: 4000
  # Optionally configure other ports to expose in the Master container
  ExtraPorts:
  # - name: BuildInfoProxy
  #   port: 9000
  # List of plugins to be install during Jenkins master start
  InstallPlugins:
    - kubernetes:1.13.7
    - workflow-job:2.30
    - workflow-aggregator:2.6
    - credentials-binding:1.17
    - git:3.9.1
    - blueocean:1.9.0
  # Used to approve a list of groovy functions in pipelines used the script-security plugin. Can be viewed under /scriptApproval
  # ScriptApproval:
  #   - "method groovy.json.JsonSlurperClassic parseText java.lang.String"
  #   - "new groovy.json.JsonSlurperClassic"
  # List of groovy init scripts to be executed during Jenkins master start
  InitScripts:
  #  - |
  #    print 'adding global pipeline libraries, register properties, bootstrap jobs...'
  # Kubernetes secret that contains a 'credentials.xml' for Jenkins
  # CredentialsXmlSecret: jenkins-credentials
  # Kubernetes secret that contains files to be put in the Jenkins 'secrets' directory,
  # useful to manage encryption keys used for credentials.xml for instance (such as
  # master.key and hudson.util.Secret)
  # SecretsFilesSecret: jenkins-secrets
  # Jenkins XML job configs to provision
  # Jobs: |-
  #   test: |-
  #     <<xml here>>
  CustomConfigMap: false
  # By default, the configMap is only used to set the initial config the first time
  # that the chart is installed.  Setting `OverwriteConfig` to `true` will overwrite
  # the jenkins config with the contents of the configMap every time the pod starts.
  OverwriteConfig: false
  # Node labels and tolerations for pod assignment
  # ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
  # ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#taints-and-tolerations-beta-feature
  NodeSelector: {}
  Tolerations: {}
  PodAnnotations: {}

  Ingress:
    ApiVersion: extensions/v1beta1
    Annotations:
      kubernetes.io/ingress.class: nginx
      kubernetes.io/tls-acme: "true"
    # Set this path to JenkinsUriPrefix above or use annotations to rewrite path
      Path: "/"
    TLS:
     - secretName: jenkins-ingress-tls
       hosts:
         - jenkins.aws.tapfeeds.com
  AdditionalConfig: {}

Agent:
  Enabled: true
  Image: jenkins/jnlp-slave
  ImageTag: latest-jdk11
  CustomJenkinsLabels: []
# ImagePullSecret: jenkins
  Component: "jenkins-slave"
  Privileged: false
  resources:
    requests:
      cpu: "200m"
      memory: "512Mi"
    limits:
      cpu: "500m"
      memory: "1024Mi"
  # You may want to change this to true while testing a new image
  AlwaysPullImage: false
  # Controls how slave pods are retained after the Jenkins build completes
  # Possible values: Always, Never, OnFailure
  PodRetention: Never
  # You can define the volumes that you want to mount for this container
  # Allowed types are: ConfigMap, EmptyDir, HostPath, Nfs, Pod, Secret
  # Configure the attributes as they appear in the corresponding Java class for that type
  # https://github.com/jenkinsci/kubernetes-plugin/tree/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/volumes
  volumes:
  # - type: Secret
  #   secretName: mysecret
  #   mountPath: /var/myapp/mysecret
  NodeSelector: {}
  # Key Value selectors. Ex:
  # jenkins-agent: v1

Persistence:
  Enabled: true
  ## A manually managed Persistent Volume and Claim
  ## Requires Persistence.Enabled: true
  ## If defined, PVC must be created manually before volume will be bound
  # ExistingClaim:

  ## jenkins data Persistent Volume Storage Class
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  # StorageClass: "-"

  Annotations: {}
  AccessMode: ReadWriteOnce
  Size: 8Gi
  volumes:
  #  - name: nothin
  #    emptyDir: {}
  mounts:
  #  - mountPath: /var/nothing
  #    name: nothing
  #    readOnly: true

NetworkPolicy:
  # Enable creation of NetworkPolicy resources.
  Enabled: false
  # For Kubernetes v1.4, v1.5 and v1.6, use 'extensions/v1beta1'
  # For Kubernetes v1.7, use 'networking.k8s.io/v1'
  ApiVersion: networking.k8s.io/v1

## Install Default RBAC roles and bindings
rbac:
  install: false
  serviceAccountName: default
  # Role reference
  roleRef: cluster-admin
  # Role kind (RoleBinding or ClusterRoleBinding)
  roleBindingKind: ClusterRoleBinding

Anything else we need to know:

kubectl logs jenkins-7bdb5b97b9-2h8bp

Error from server (BadRequest): container "jenkins" in pod "jenkins-7bdb5b97b9-2h8bp" is waiting to start: PodInitializing

kubectl logs jenkins-7bdb5b97b9-2h8bp -p

Error from server (BadRequest): previous terminated container "jenkins" in pod "jenkins-7bdb5b97b9-2h8bp" not found

kubectl describe pod jenkins-7bdb5b97b9-2h8bp

--

Name:           jenkins-7bdb5b97b9-2h8bp
Namespace:      kube-system
Node:           ip-172-20-151-70.cn-northwest-1.compute.internal/172.20.151.70
Start Time:     Mon, 10 Dec 2018 14:45:29 +0800
Labels:         app=jenkins
                chart=jenkins-0.25.0
                component=jenkins-jenkins-master
                heritage=Tiller
                pod-template-hash=3686165365
                release=jenkins
Annotations:    checksum/config: e94dd0b017b820686f611035f38940ea53c172af104b0ee8da928e068a5966e8
Status:         Pending
IP:             100.96.5.13
Controlled By:  ReplicaSet/jenkins-7bdb5b97b9
Init Containers:
  copy-default-config:
    Container ID:  docker://c4dd267ad6c5400caba29f1aa1ff8f5b8fba2c2c6bb573ac4e9bb8bc2bc67cb7
    Image:         jenkins/jenkins:jdk11
    Image ID:      docker-pullable://jenkins/jenkins@sha256:cb7dfc139faf74eb37c860600ccd4a8c8df683699bc80db4b1766873c20de0c9
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      /var/jenkins_config/apply_config.sh
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 10 Dec 2018 15:16:16 +0800
      Finished:     Mon, 10 Dec 2018 15:19:04 +0800
    Ready:          False
    Restart Count:  7
    Limits:
      cpu:     2
      memory:  2Gi
    Requests:
      cpu:        500m
      memory:     512Mi
    Environment:  <none>
    Mounts:
      /usr/share/jenkins/ref/secrets/ from secrets-dir (rw)
      /var/jenkins_config from jenkins-config (rw)
      /var/jenkins_home from jenkins-home (rw)
      /var/jenkins_plugins from plugin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-slv79 (ro)
Containers:
  jenkins:
    Container ID:  
    Image:         jenkins/jenkins:jdk11
    Image ID:      
    Ports:         8080/TCP, 50000/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      --argumentsRealm.passwd.$(ADMIN_USER)=$(ADMIN_PASSWORD)
      --argumentsRealm.roles.$(ADMIN_USER)=admin
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  2Gi
    Requests:
      cpu:      500m
      memory:   512Mi
    Liveness:   http-get http://:http/login delay=90s timeout=5s period=10s #success=1 #failure=12
    Readiness:  http-get http://:http/login delay=60s timeout=1s period=10s #success=1 #failure=3
    Environment:
      JAVA_TOOL_OPTIONS:  -Xms512m -Xmx1024m
      JENKINS_OPTS:       
      ADMIN_PASSWORD:     <set to the key 'jenkins-admin-password' in secret 'jenkins'>  Optional: false
      ADMIN_USER:         <set to the key 'jenkins-admin-user' in secret 'jenkins'>      Optional: false
    Mounts:
      /usr/share/jenkins/ref/plugins/ from plugin-dir (rw)
      /usr/share/jenkins/ref/secrets/ from secrets-dir (rw)
      /var/jenkins_config from jenkins-config (ro)
      /var/jenkins_home from jenkins-home (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-slv79 (ro)
Conditions:
  Type           Status
  Initialized    False 
  Ready          False 
  PodScheduled   True 
Volumes:
  jenkins-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      jenkins
    Optional:  false
  plugin-dir:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
  secrets-dir:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
  jenkins-home:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  jenkins
    ReadOnly:   false
  default-token-slv79:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-slv79
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age                 From                                                       Message
  ----     ------                  ----                ----                                                       -------
  Normal   SuccessfulMountVolume   35m                 kubelet, ip-172-20-151-70.cn-northwest-1.compute.internal  MountVolume.SetUp succeeded for volume "plugin-dir"
  Normal   SuccessfulMountVolume   35m                 kubelet, ip-172-20-151-70.cn-northwest-1.compute.internal  MountVolume.SetUp succeeded for volume "secrets-dir"
  Normal   SuccessfulMountVolume   35m                 kubelet, ip-172-20-151-70.cn-northwest-1.compute.internal  MountVolume.SetUp succeeded for volume "jenkins-config"
  Normal   SuccessfulMountVolume   35m                 kubelet, ip-172-20-151-70.cn-northwest-1.compute.internal  MountVolume.SetUp succeeded for volume "default-token-slv79"
  Normal   SuccessfulAttachVolume  35m                 attachdetach-controller                                    AttachVolume.Attach succeeded for volume "pvc-246ced27-fc43-11e8-bf48-0204830a75be"
  Normal   Scheduled               35m                 default-scheduler                                          Successfully assigned jenkins-7bdb5b97b9-2h8bp to ip-172-20-151-70.cn-northwest-1.compute.internal
  Normal   SuccessfulMountVolume   35m                 kubelet, ip-172-20-151-70.cn-northwest-1.compute.internal  MountVolume.SetUp succeeded for volume "pvc-246ced27-fc43-11e8-bf48-0204830a75be"
  Normal   Pulling                 26m (x4 over 35m)   kubelet, ip-172-20-151-70.cn-northwest-1.compute.internal  pulling image "jenkins/jenkins:jdk11"
  Normal   Pulled                  26m (x4 over 35m)   kubelet, ip-172-20-151-70.cn-northwest-1.compute.internal  Successfully pulled image "jenkins/jenkins:jdk11"
  Normal   Created                 26m (x4 over 35m)   kubelet, ip-172-20-151-70.cn-northwest-1.compute.internal  Created container
  Normal   Started                 26m (x4 over 35m)   kubelet, ip-172-20-151-70.cn-northwest-1.compute.internal  Started container
  Warning  BackOff                 57s (x57 over 30m)  kubelet, ip-172-20-151-70.cn-northwest-1.compute.internal  Back-off restarting failed container
-- Zhuang Niu
jenkins
kubernetes
kubernetes-helm

2 Answers

5/15/2019

You could try to set hostNetworking: true. It should help.)

-- Eugene Lopatkin
Source: StackOverflow

12/11/2018

I figure out this issue by below steps:

  1. ssh k8s node
  2. journalctl -fu docker.service

returned error: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting.

gcr.io is a google website, which is blocked in China. I need a VPN to walk it through.

-- Zhuang Niu
Source: StackOverflow