Jenkins on Kubernetes not loading /var/jenkins_home/init.groovy.d files in external volumes

6/7/2017

I am new to Kubernetes, While I am creating a Jenkins pod in GKE, /var/jenkins_home/init.groovy.d files are not loading(Only tcp-slave-agent-port.groovy file is loaded). I have created my jenkins image (for populating plugins and uploading groovy script) from the official image. And if I am running the deployment without volume attachment, everything will works fine. Anyone can help me please?

My Docker file:

FROM jenkins
MAINTAINER Bujail
# Install plugins
COPY plugins.txt /usr/share/jenkins/plugins
RUN /usr/local/bin/install-plugins.sh $(cat /usr/share/jenkins/plugins | tr '\n' ' ')
# Setup Security with User
COPY security.groovy /var/jenkins_home/init.groovy.d/security.groovy
# Disabling setup wizard
ENV JAVA_OPTS="-Djenkins.install.runSetupWizard=false -Djenkins.CLI.disabled=true"

Groovy Script:

#!groovy
import jenkins.model.*
import hudson.security.*
import jenkins.security.s2m.AdminWhitelistRule

def instance = Jenkins.getInstance()

println "--> creating local user 'admin'"

def hudsonRealm = new HudsonPrivateSecurityRealm(false)
hudsonRealm.createAccount('admin','admin@123')
instance.setSecurityRealm(hudsonRealm)

def strategy = new
hudson.security.FullControlOnceLoggedInAuthorizationStrategy()
strategy.setAllowAnonymousRead(false)
instance.setAuthorizationStrategy(strategy)

println "--> Enable Agent → Master Access Control"
Jenkins.instance.getInjector().getInstance(AdminWhitelistRule.class)
.setMasterKillSwitch(false);
instance.save()

Plugins.txt

maven-plugin:2.15.1
bitbucket:1.1.5
artifactory:2.10.3
sonar:2.6.1
kubernetes:0.11

Kubernetes Deployment file:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins
  namespace: immediate
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: master
    spec:
      containers:
      - name: master
        image: bujail/private:jenkins
        ports:
        - containerPort: 8080
        - containerPort: 50000
        readinessProbe:
          httpGet:
            path: /login
            port: 8080
          periodSeconds: 10
          timeoutSeconds: 5
          successThreshold: 2
          failureThreshold: 5
        volumeMounts:
        - name: jenkins-home
          mountPath: /var/jenkins_home
        resources:
          limits:
            cpu: 500m
            memory: 1500Mi
          requests:
            cpu: 500m
            memory: 1500Mi
      securityContext:
        fsGroup: 1000
        seLinuxOptions:
          level: "s0:c123,c456"         
      imagePullSecrets:
      - name: docker-buju
      volumes:
      - name: jenkins-home
        persistentVolumeClaim:
          claimName: jenkins

StorageClass:

apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: jenkins
  namespace: immediate
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd
  zone: asia-east1-c

PersistantVolumeClaim:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: jenkins
  namespace: immediate
  annotations:
    pv.beta.kubernetes.io/gid: "1000"
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: jenkins

And Jenkins will load without security enabled. If I try to enable security manually, I won't persistent.

kubectl logs :

Running from: /usr/share/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
Jun 07, 2017 8:45:03 AM Main deleteWinstoneTempContents
WARNING: Failed to delete the temporary Winstone file /tmp/winstone/jenkins.war
Jun 07, 2017 8:45:03 AM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: Logging initialized @2087ms
Jun 07, 2017 8:45:03 AM winstone.Logger logInternal
INFO: Beginning extraction from war file
Jun 07, 2017 8:45:07 AM org.eclipse.jetty.util.log.JavaUtilLog warn
WARNING: Empty contextPath
Jun 07, 2017 8:45:08 AM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: jetty-9.2.z-SNAPSHOT
Jun 07, 2017 8:45:11 AM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
Jenkins home directory: /var/jenkins_home found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
Jun 07, 2017 8:45:14 AM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: Started w.@3e34ace1{/,file:/var/jenkins_home/war/,AVAILABLE}{/var/jenkins_home/war}
Jun 07, 2017 8:45:14 AM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: Started ServerConnector@5778826f{HTTP/1.1}{0.0.0.0:8080}
Jun 07, 2017 8:45:14 AM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: Started @12793ms
Jun 07, 2017 8:45:14 AM winstone.Logger logInternal
INFO: Winstone Servlet Engine v2.0 running: controlPort=disabled
Jun 07, 2017 8:45:17 AM jenkins.InitReactorRunner$1 onAttained
INFO: Started initialization
Jun 07, 2017 8:45:38 AM jenkins.InitReactorRunner$1 onAttained
INFO: Listed all plugins
Jun 07, 2017 8:45:55 AM jenkins.InitReactorRunner$1 onAttained
INFO: Prepared all plugins
Jun 07, 2017 8:46:02 AM jenkins.InitReactorRunner$1 onAttained
INFO: Started all plugins
Jun 07, 2017 8:46:02 AM jenkins.InitReactorRunner$1 onAttained
INFO: Augmented all extensions
Jun 07, 2017 8:46:02 AM jenkins.InitReactorRunner$1 onAttained
INFO: Loaded all jobs
Jun 07, 2017 8:46:03 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started Download metadata
Jun 07, 2017 8:46:04 AM jenkins.util.groovy.GroovyHookScript execute
INFO: Executing /var/jenkins_home/init.groovy.d/tcp-slave-agent-port.groovy
Jun 07, 2017 8:46:06 AM jenkins.InitReactorRunner$1 onAttained
INFO: Completed initialization
Jun 07, 2017 8:46:06 AM hudson.WebAppMain$3 run
INFO: Jenkins is fully up and running
--> setting agent port for jnlp
Jun 07, 2017 8:46:18 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted connection #1 from /10.20.1.21:59828
--> setting agent port for jnlp... done
Jun 07, 2017 8:46:28 AM hudson.model.UpdateSite updateData
INFO: Obtained the latest update center data file for UpdateSource default
Jun 07, 2017 8:46:30 AM hudson.model.DownloadService$Downloadable load
INFO: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller
Jun 07, 2017 8:46:31 AM hudson.model.DownloadService$Downloadable load
INFO: Obtained the updated data file for hudson.tasks.Ant.AntInstaller
Jun 07, 2017 8:46:32 AM hudson.model.DownloadService$Downloadable load
INFO: Obtained the updated data file for hudson.plugins.gradle.GradleInstaller
Jun 07, 2017 8:46:33 AM hudson.model.DownloadService$Downloadable load
INFO: Obtained the updated data file for hudson.plugins.sonar.MsBuildSonarQubeRunnerInstaller
Jun 07, 2017 8:46:34 AM hudson.model.DownloadService$Downloadable load
INFO: Obtained the updated data file for hudson.plugins.sonar.SonarRunnerInstaller
Jun 07, 2017 8:46:40 AM hudson.model.DownloadService$Downloadable load
INFO: Obtained the updated data file for hudson.tools.JDKInstaller
Jun 07, 2017 8:46:40 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished Download metadata. 36,995 ms
Jun 07, 2017 3:36:21 PM org.eclipse.jetty.util.log.JavaUtilLog warn
WARNING: Illegal character 0x4 in state=START for buffer HeapByteBuffer@1c2e6542[p=1,l=10,c=16384,r=9]={\x04<<<\x01\x00P_\xD5\xB1|0\x00>>> HTTP/1.1\r\nHost: ...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
Jun 07, 2017 3:36:22 PM org.eclipse.jetty.util.log.JavaUtilLog warn
WARNING: badMessage: 400 Illegal character 0x4 for HttpChannelOverHttp@479b05c8{r=0,c=false,a=IDLE,uri=}
-- bujail
google-cloud-datastore
google-cloud-platform
google-kubernetes-engine
jenkins

1 Answer

6/19/2017

A Big lesson I have learned. If we are mounting a volume to an existing folder inside an Image(Container), It will replace all content with the attached volume's content. For resolving my issue, I have edited my Dockerfile to copy groovy script to /usr/share/jenkins/ref/init.groovy.d/ . While loading container, startup script will copy whole files from this folder to jenkins home!

-- bujail
Source: StackOverflow