kubernetes jenkins plugin does not create 2 containers

4/16/2020

I have 2 Jenkins instances, one use version 1.8 and second version 1.18.

Oldest version is able to create both containers.

Agent specification [Kubernetes Pod Template] (mo-aio-build-supplier): 
* [jnlp] mynexus.services.com/mo-base/jenkins-slave-mo-aio:1.8.2-ca(resourceRequestCpu: 0.25, resourceRequestMemory: 256Mi, resourceLimitCpu: 1, resourceLimitMemory: 1.5Gi)
* [postgres] mynexus.services.com:443/mo-base/mo-base-postgresql-95-openshift

Newest version are not able to create postgres container

Container postgres exited with error 1. Logs: mkdir: cannot create directory '/home/jenkins': Permission denied

Both use same podTemplate

 podTemplate(
            name: label,
            label: label,
            cloud: 'openshift',
            serviceAccount: 'jenkins',
            containers: [
                    containerTemplate(
                            name: 'jnlp',
                            image: 'mynexus.services.theosmo.com/jenkins-slave-mo-aio:v3.11.104-14_jdk8',
                            resourceRequestCpu: env.CPU_REQUEST,
                            resourceLimitCpu: env.CPU_LIMIT,
                            resourceRequestMemory: env.RAM_REQUEST,
                            resourceLimitMemory: env.RAM_LIMIT,
                            workingDir: '/tmp',
                            args: '${computer.jnlpmac} ${computer.name}',
                            command: ''
                    ),
                    containerTemplate(
                            name: 'postgres',
                            image: 'mynexus.services.theosmo.com:443/mo-base/mo-base-postgresql-95-openshift',
                            envVars: [
                                envVar(key: "POSTGRESQL_USER", value: "admin"),
                                envVar(key: "POSTGRESQL_PASSWORD", value: "admin"),
                                envVar(key: "POSTGRESQL_DATABASE", value: "supplier_data"),
                            ]
                    )
            ],
            volumes: [emptyDirVolume(mountPath: '/dev/shm', memory: true)]
    )

Also, I've noticed YAML created by newest version is a bit weird

apiVersion: "v1"
kind: "Pod"
metadata:
  annotations:
    buildUrl: "http://jenkins.svc:80/job/build-supplier/473/"
  labels:
    jenkins: "slave"
    jenkins/mo-aio-build-supplier: "true"
  name: "mo-aio-build-supplier-xfgmn-qmrdl"
spec:
  containers:
  - args:
    - "********"
    - "mo-aio-build-supplier-xfgmn-qmrdl"
    env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "jenkins-jnlp.svc:50000"
    - name: "JENKINS_AGENT_NAME"
      value: "mo-aio-build-supplier-xfgmn-qmrdl"
    - name: "JENKINS_NAME"
      value: "mo-aio-build-supplier-xfgmn-qmrdl"
    - name: "JENKINS_AGENT_WORKDIR"
      value: "/tmp"
    - name: "JENKINS_URL"
      value: "http://jenkins.svc:80/"
    - name: "HOME"
      value: "/home/jenkins"
    image: "mynexus.services.com/mo-base/jenkins-slave-mo-aio:1.8.2-ca"
    imagePullPolicy: "IfNotPresent"
    name: "jnlp"
    resources:
      limits:
        memory: "1.5Gi"
        cpu: "1"
      requests:
        memory: "256Mi"
        cpu: "0.25"
    securityContext:
      privileged: false
    tty: false
    volumeMounts:
    - mountPath: "/dev/shm"
      name: "volume-0"
      readOnly: false
    - mountPath: "/tmp"
      name: "workspace-volume"
      readOnly: false
    workingDir: "/tmp"
  - env:
    - name: "POSTGRESQL_DATABASE"
      value: "supplier_data"
    - name: "POSTGRESQL_USER"
      value: "admin"
    - name: "HOME"
      value: "/home/jenkins"
    - name: "POSTGRESQL_PASSWORD"
      value: "admin"
    image: "mynexus.services.com:443/mo-base/mo-base-postgresql-95-openshift"
    imagePullPolicy: "IfNotPresent"
    name: "postgres"
    resources:
      limits: {}
      requests: {}
    securityContext:
      privileged: false
    tty: false
    volumeMounts:
    - mountPath: "/dev/shm"
      name: "volume-0"
      readOnly: false
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
      readOnly: false
    workingDir: "/home/jenkins/agent"
  nodeSelector: {}
  restartPolicy: "Never"
  serviceAccount: "jenkins"
  volumes:
  - emptyDir:
      medium: "Memory"
    name: "volume-0"
  - emptyDir: {}
    name: "workspace-volume"

As you are able to see above:

  • postgres container is under an env tree

Any suggestion? Thanks in advance

-- Rafael Ruiz Tabares
jenkins
kubernetes
kubernetes-pod

1 Answer

4/16/2020

As far as I checked there

The problem

Since Kubernetes Plugin version 1.18.0, the default working directory of the pod containers was changed from /home/jenkins to /home/jenkins/agent. But the default HOME environment variable enforcement is still pointing to /home/jenkins. The impact of this change is that if pod container images do not have a /home/jenkins directory with sufficient permissions for the running user, builds will fail to do anything directly under their HOME directory, /home/jenkins.

Resolution

There are different workaround to that problem:

Change the default HOME variable

The simplest and preferred workaround is to add the system property -Dorg.csanchez.jenkins.plugins.kubernetes.PodTemplateBuilder.defaultHome=/home/jenkins/agent on Jenkins startup. This requires a restart.

This workaround will reflect the behavior of kubernetes plugin pre-1.18.0 but on the new working directory /home/jenkins/agent

Use /home/jenkins as the working directory

A workaround is to change the working directory of pod containers back to /home/jenkins. This workaround is only possible when using YAML to define agent pod templates (see JENKINS-60977).

Prepare images for Jenkins

A workaround could be to ensure that the images used in agent pods have a /home/jenkins directory that is owned by the root group and writable by the root group as mentioned in OpenShift Container Platform-specific guidelines.


Additionaly there is the issue on jenkins.


Hope this helps.

-- jt97
Source: StackOverflow