K
Q

Question

Jenkins jnlp slave not working on local Kubernetes cluster

6/14/2019

I have tried local minikube and kubeadm Kubernetes cluster and ran Jenkins server through https://github.com/jenkinsci/kubernetes-plugin and configured JNLP slave using.

Following are the more details to understand the issue.

Kubernetes pod template as below.
The Jenkins job is configured with Github Organization plugin.
Jenkinsfile is like below in Github repo.
node ('slave'){ stage('Checkout') { checkout scm } stage('Build Docker image') { sh 'make build' } }
When I run the Jenkins job, slave is starting but it is not running anything from Jenkinfile instead it just shows following logs in Jenkins job console result.
[Pipeline] Start of Pipeline [Pipeline] node Still waiting to schedule task ‘Jenkins’ doesn’t have label ‘slave’
At the same time Jenkins job is still running and I can see following logs. Here it shows that Jenkins launched slave and it ran for around 105 seconds and got terminated without running anything from Jenkinsfile and it starts another slave again and again.
$ kubectl get pods --watch NAME READY STATUS RESTARTS AGE jenkins-0 1/1 Running 0 45h slave-bls6d 0/1 Pending 0 0s slave-bls6d 0/1 Pending 0 0s slave-bls6d 0/1 ContainerCreating 0 0s slave-bls6d 0/1 ContainerCreating 0 2s slave-bls6d 1/1 Running 0 4s slave-bls6d 1/1 Terminating 0 105s slave-bls6d 1/1 Terminating 0 105s slave-jdq9l 0/1 Pending 0 0s slave-jdq9l 0/1 Pending 0 0s slave-jdq9l 0/1 ContainerCreating 0 0s slave-jdq9l 0/1 ContainerCreating 0 2s slave-jdq9l 1/1 Running 0 5s slave-jdq9l 1/1 Terminating 0 105s slave-jdq9l 1/1 Terminating 0 106s slave-06dfz 0/1 Pending 0 0s slave-06dfz 0/1 Pending 0 0s slave-06dfz 0/1 ContainerCreating 0 0s slave-06dfz 0/1 ContainerCreating 0 3s slave-06dfz 1/1 Running 0 4s slave-06dfz 1/1 Terminating 0 105s slave-06dfz 1/1 Terminating 0 105s

At the same time I can see these logs in kubernetes events.

$ kubectl get events --watch
0s    Normal   Scheduled   Pod   Successfully assigned kubernetes-plugin/slave-bls6d to ubuntu
0s    Normal   Pulled   Pod   Container image "jenkins/jnlp-slave" already present on machine
0s    Normal   Created   Pod   Created container jnlp
0s    Normal   Started   Pod   Started container jnlp
0s    Normal   Killing   Pod   Stopping container jnlp
0s    Normal   Killing   Pod   Stopping container jnlp
0s    Normal   Scheduled   Pod   Successfully assigned kubernetes-plugin/slave-jdq9l to ubuntu
0s    Normal   Pulled   Pod   Container image "jenkins/jnlp-slave" already present on machine
0s    Normal   Created   Pod   Created container jnlp
0s    Normal   Started   Pod   Started container jnlp
0s    Normal   Killing   Pod   Stopping container jnlp
0s    Normal   Killing   Pod   Stopping container jnlp
0s    Normal   Scheduled   Pod   Successfully assigned kubernetes-plugin/slave-06dfz to ubuntu
0s    Normal   Pulled   Pod   Container image "jenkins/jnlp-slave" already present on machine
0s    Normal   Created   Pod   Created container jnlp
0s    Normal   Started   Pod   Started container jnlp
0s    Normal   Killing   Pod   Stopping container jnlp
0s    Normal   Killing   Pod   Stopping container jnlp
0s    Normal   Scheduled   Pod   Successfully assigned kubernetes-plugin/slave-5f3p6 to ubuntu
0s    Normal   Pulled   Pod   Container image "jenkins/jnlp-slave" already present on machine
0s    Normal   Created   Pod   Created container jnlp
0s    Normal   Started   Pod   Started container jnlp
0s    Normal   Killing   Pod   Stopping container jnlp
1s    Normal   Killing   Pod   Stopping container jnlp
0s    Normal   Scheduled   Pod   Successfully assigned kubernetes-plugin/slave-nw32h to ubuntu
0s    Normal   Pulled   Pod   Container image "jenkins/jnlp-slave" already present on machine
0s    Normal   Created   Pod   Created container jnlp
1s    Normal   Started   Pod   Started container jnlp
0s    Normal   Killing   Pod   Stopping container jnlp
0s    Normal   Killing   Pod   Stopping container jnlp
0s    Normal   Scheduled   Pod   Successfully assigned kubernetes-plugin/slave-l5jn5 to ubuntu
0s    Normal   Pulled   Pod   Container image "jenkins/jnlp-slave" already present on machine
0s    Normal   Created   Pod   Created container jnlp
0s    Normal   Started   Pod   Started container jnlp
0s    Normal   Killing   Pod   Stopping container jnlp
0s    Normal   Killing   Pod   Stopping container jnlp

Configuration wise everything looks fine to me but why is it now running defined jobs from Jenkinsfile. Why the slave is just hanging around and getting terminated after 105 seconds instead of running the job.

edit 1: I have enabled Jenkins log recorder for org.csanchez.jenkins.plugins.kubernetes at ALL level and getting following logs.

Jun 17, 2019 2:49:28 PM FINE org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud

In provisioning : []

Jun 17, 2019 2:49:28 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision

Excess workload after pending Kubernetes agents: 1

Jun 17, 2019 2:49:28 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision

Template for label slave: Kubernetes Pod Template

Jun 17, 2019 2:49:28 PM FINE org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud

Building connection to Kubernetes kubernetes URL https://kubernetes.default.svc.cluster.local namespace kubernetes-plugin

Jun 17, 2019 2:49:28 PM FINE org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud

Connected to Kubernetes kubernetes URL https://kubernetes.default.svc.cluster.local/

Jun 17, 2019 2:49:28 PM FINEST org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud

Planned Kubernetes agents for template "Kubernetes Pod Template": 1

Jun 17, 2019 2:49:38 PM FINE org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud

Building connection to Kubernetes kubernetes URL https://kubernetes.default.svc.cluster.local namespace kubernetes-plugin

Jun 17, 2019 2:49:38 PM FINE org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud

Connected to Kubernetes kubernetes URL https://kubernetes.default.svc.cluster.local/

Jun 17, 2019 2:49:38 PM FINE org.csanchez.jenkins.plugins.kubernetes.PodTemplateBuilder

Adding workspace volume from template: EmptyDirWorkspaceVolume [memory=false]

Jun 17, 2019 2:49:38 PM FINE org.csanchez.jenkins.plugins.kubernetes.PodTemplateBuilder

Pod built: Pod(apiVersion=v1, kind=Pod, metadata=ObjectMeta(annotations={}, clusterName=null, creationTimestamp=null, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, initializers=null, labels={jenkins=slave, jenkins/slave=true}, name=slave-738cb, namespace=null, ownerReferences=[], resourceVersion=null, selfLink=null, uid=null, additionalProperties={}), spec=PodSpec(activeDeadlineSeconds=null, affinity=null, automountServiceAccountToken=null, containers=[Container(args=[cat], command=[/bin/sh, -c], env=[EnvVar(name=JENKINS_SECRET, value=0cfd3e63aec16128455251edad357882243691c9f90a7a2e022eea572c6bc690, valueFrom=null, additionalProperties={}), EnvVar(name=JENKINS_AGENT_NAME, value=slave-738cb, valueFrom=null, additionalProperties={}), EnvVar(name=JENKINS_NAME, value=slave-738cb, valueFrom=null, additionalProperties={}), EnvVar(name=JENKINS_URL, value=http://jenkins.example.com/, valueFrom=null, additionalProperties={}), EnvVar(name=HOME, value=/home/jenkins, valueFrom=null, additionalProperties={})], envFrom=[], image=jenkins/jnlp-slave, imagePullPolicy=IfNotPresent, lifecycle=null, livenessProbe=null, name=jnlp, ports=[], readinessProbe=null, resources=ResourceRequirements(limits={}, requests={}, additionalProperties={}), securityContext=SecurityContext(allowPrivilegeEscalation=null, capabilities=null, privileged=true, procMount=null, readOnlyRootFilesystem=null, runAsGroup=null, runAsNonRoot=null, runAsUser=null, seLinuxOptions=null, additionalProperties={}), stdin=null, stdinOnce=null, terminationMessagePath=null, terminationMessagePolicy=null, tty=true, volumeDevices=[], volumeMounts=[VolumeMount(mountPath=/home/jenkins, mountPropagation=null, name=workspace-volume, readOnly=false, subPath=null, subPathExpr=null, additionalProperties={})], workingDir=/home/jenkins, additionalProperties={})], dnsConfig=null, dnsPolicy=null, enableServiceLinks=null, hostAliases=[], hostIPC=null, hostNetwork=null, hostPID=null, hostname=null, imagePullSecrets=[], initContainers=[], nodeName=null, nodeSelector={}, priority=null, priorityClassName=null, readinessGates=[], restartPolicy=Never, runtimeClassName=null, schedulerName=null, securityContext=null, serviceAccount=null, serviceAccountName=null, shareProcessNamespace=null, subdomain=null, terminationGracePeriodSeconds=null, tolerations=[], volumes=[Volume(awsElasticBlockStore=null, azureDisk=null, azureFile=null, cephfs=null, cinder=null, configMap=null, csi=null, downwardAPI=null, emptyDir=EmptyDirVolumeSource(medium=, sizeLimit=null, additionalProperties={}), fc=null, flexVolume=null, flocker=null, gcePersistentDisk=null, gitRepo=null, glusterfs=null, hostPath=null, iscsi=null, name=workspace-volume, nfs=null, persistentVolumeClaim=null, photonPersistentDisk=null, portworxVolume=null, projected=null, quobyte=null, rbd=null, scaleIO=null, secret=null, storageos=null, vsphereVolume=null, additionalProperties={})], additionalProperties={}), status=null, additionalProperties={})

Jun 17, 2019 2:49:38 PM FINE org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher

Creating Pod: kubernetes-plugin/slave-738cb

Jun 17, 2019 2:49:38 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Created Pod: kubernetes-plugin/slave-738cb

Jun 17, 2019 2:49:38 PM FINEST org.csanchez.jenkins.plugins.kubernetes.AllContainersRunningPodWatcher

[ADDED] slave-738cb

Jun 17, 2019 2:49:40 PM FINEST org.csanchez.jenkins.plugins.kubernetes.AllContainersRunningPodWatcher

[MODIFIED] slave-738cb

Jun 17, 2019 2:49:42 PM FINEST org.csanchez.jenkins.plugins.kubernetes.AllContainersRunningPodWatcher

[MODIFIED] slave-738cb

Jun 17, 2019 2:49:42 PM FINE org.csanchez.jenkins.plugins.kubernetes.AllContainersRunningPodWatcher

All containers are running for pod slave-738cb

Jun 17, 2019 2:49:42 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Pod is running: kubernetes-plugin/slave-738cb

Jun 17, 2019 2:49:42 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (0/100): slave-738cb

Jun 17, 2019 2:49:43 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (1/100): slave-738cb

Jun 17, 2019 2:49:44 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (2/100): slave-738cb

Jun 17, 2019 2:49:45 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (3/100): slave-738cb

Jun 17, 2019 2:49:46 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (4/100): slave-738cb

Jun 17, 2019 2:49:47 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (5/100): slave-738cb

Jun 17, 2019 2:49:48 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (6/100): slave-738cb

Jun 17, 2019 2:49:49 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (7/100): slave-738cb

Jun 17, 2019 2:49:50 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (8/100): slave-738cb

Jun 17, 2019 2:49:51 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (9/100): slave-738cb

Jun 17, 2019 2:49:52 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (10/100): slave-738cb

Jun 17, 2019 2:49:53 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (11/100): slave-738cb

Jun 17, 2019 2:49:54 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (12/100): slave-738cb

Jun 17, 2019 2:49:55 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (13/100): slave-738cb

Jun 17, 2019 2:49:56 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (14/100): slave-738cb

Jun 17, 2019 2:49:57 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (15/100): slave-738cb

Jun 17, 2019 2:49:58 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (16/100): slave-738cb

Jun 17, 2019 2:49:59 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (17/100): slave-738cb

Jun 17, 2019 2:50:00 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (18/100): slave-738cb

Jun 17, 2019 2:50:01 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (19/100): slave-738cb

Jun 17, 2019 2:50:02 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (20/100): slave-738cb

Jun 17, 2019 2:50:03 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (21/100): slave-738cb

Jun 17, 2019 2:50:04 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (22/100): slave-738cb

Jun 17, 2019 2:50:05 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (23/100): slave-738cb

Jun 17, 2019 2:50:06 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (24/100): slave-738cb

Jun 17, 2019 2:50:07 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (25/100): slave-738cb

Jun 17, 2019 2:50:08 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (26/100): slave-738cb

Jun 17, 2019 2:50:09 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (27/100): slave-738cb

Jun 17, 2019 2:50:10 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (28/100): slave-738cb

Jun 17, 2019 2:50:11 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (29/100): slave-738cb

Jun 17, 2019 2:50:12 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (30/100): slave-738cb

Jun 17, 2019 2:50:13 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (31/100): slave-738cb

Jun 17, 2019 2:50:14 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (32/100): slave-738cb

Jun 17, 2019 2:50:15 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (33/100): slave-738cb

Jun 17, 2019 2:50:16 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (34/100): slave-738cb

Jun 17, 2019 2:50:17 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (35/100): slave-738cb

Jun 17, 2019 2:50:18 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (36/100): slave-738cb

Jun 17, 2019 2:50:20 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (37/100): slave-738cb

Jun 17, 2019 2:50:21 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (38/100): slave-738cb

Jun 17, 2019 2:50:22 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (39/100): slave-738cb

Jun 17, 2019 2:50:23 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (40/100): slave-738cb

Jun 17, 2019 2:50:24 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (41/100): slave-738cb

Jun 17, 2019 2:50:25 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (42/100): slave-738cb

Jun 17, 2019 2:50:26 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (43/100): slave-738cb

Jun 17, 2019 2:50:27 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (44/100): slave-738cb

Jun 17, 2019 2:50:28 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (45/100): slave-738cb

Jun 17, 2019 2:50:29 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (46/100): slave-738cb

Jun 17, 2019 2:50:30 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (47/100): slave-738cb

Jun 17, 2019 2:50:31 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (48/100): slave-738cb

Jun 17, 2019 2:50:32 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (49/100): slave-738cb

Jun 17, 2019 2:50:33 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (50/100): slave-738cb

Jun 17, 2019 2:50:34 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (51/100): slave-738cb

Jun 17, 2019 2:50:35 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (52/100): slave-738cb

Jun 17, 2019 2:50:36 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (53/100): slave-738cb

Jun 17, 2019 2:50:37 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (54/100): slave-738cb

Jun 17, 2019 2:50:38 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (55/100): slave-738cb

Jun 17, 2019 2:50:39 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (56/100): slave-738cb

Jun 17, 2019 2:50:40 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (57/100): slave-738cb

Jun 17, 2019 2:50:41 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (58/100): slave-738cb

Jun 17, 2019 2:50:42 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (59/100): slave-738cb

Jun 17, 2019 2:50:43 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (60/100): slave-738cb

Jun 17, 2019 2:50:44 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (61/100): slave-738cb

Jun 17, 2019 2:50:45 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (62/100): slave-738cb

Jun 17, 2019 2:50:46 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (63/100): slave-738cb

Jun 17, 2019 2:50:47 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (64/100): slave-738cb

Jun 17, 2019 2:50:48 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (65/100): slave-738cb

Jun 17, 2019 2:50:49 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (66/100): slave-738cb

Jun 17, 2019 2:50:50 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (67/100): slave-738cb

Jun 17, 2019 2:50:51 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (68/100): slave-738cb

Jun 17, 2019 2:50:52 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (69/100): slave-738cb

Jun 17, 2019 2:50:53 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (70/100): slave-738cb

Jun 17, 2019 2:50:54 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (71/100): slave-738cb

Jun 17, 2019 2:50:55 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (72/100): slave-738cb

Jun 17, 2019 2:50:56 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (73/100): slave-738cb

Jun 17, 2019 2:50:57 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (74/100): slave-738cb

Jun 17, 2019 2:50:58 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (75/100): slave-738cb

Jun 17, 2019 2:50:59 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (76/100): slave-738cb

Jun 17, 2019 2:51:00 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (77/100): slave-738cb

Jun 17, 2019 2:51:01 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (78/100): slave-738cb

Jun 17, 2019 2:51:02 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (79/100): slave-738cb

Jun 17, 2019 2:51:03 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (80/100): slave-738cb

Jun 17, 2019 2:51:04 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (81/100): slave-738cb

Jun 17, 2019 2:51:05 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (82/100): slave-738cb

Jun 17, 2019 2:51:06 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (83/100): slave-738cb

Jun 17, 2019 2:51:07 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (84/100): slave-738cb

Jun 17, 2019 2:51:08 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (85/100): slave-738cb

Jun 17, 2019 2:51:09 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (86/100): slave-738cb

Jun 17, 2019 2:51:10 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (87/100): slave-738cb

Jun 17, 2019 2:51:11 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (88/100): slave-738cb

Jun 17, 2019 2:51:12 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (89/100): slave-738cb

Jun 17, 2019 2:51:13 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (90/100): slave-738cb

Jun 17, 2019 2:51:14 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (91/100): slave-738cb

Jun 17, 2019 2:51:15 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (92/100): slave-738cb

Jun 17, 2019 2:51:16 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (93/100): slave-738cb

Jun 17, 2019 2:51:17 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (94/100): slave-738cb

Jun 17, 2019 2:51:18 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (95/100): slave-738cb

Jun 17, 2019 2:51:19 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (96/100): slave-738cb

Jun 17, 2019 2:51:20 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (97/100): slave-738cb

Jun 17, 2019 2:51:21 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (98/100): slave-738cb

Jun 17, 2019 2:51:22 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Waiting for agent to connect (99/100): slave-738cb

Jun 17, 2019 2:51:23 PM WARNING org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch

Error in provisioning; agent=KubernetesSlave name: slave-738cb, template=PodTemplate{inheritFrom='', name='slave', namespace='', label='slave', nodeSelector='', nodeUsageMode=EXCLUSIVE, workspaceVolume=EmptyDirWorkspaceVolume [memory=false], containers=[ContainerTemplate{name='jnlp', image='jenkins/jnlp-slave', privileged=true, workingDir='/home/jenkins', command='/bin/sh -c', args='cat', ttyEnabled=true, resourceRequestCpu='', resourceRequestMemory='', resourceLimitCpu='', resourceLimitMemory='', livenessProbe=org.csanchez.jenkins.plugins.kubernetes.ContainerLivenessProbe@7f0aa5f6}]}
java.lang.IllegalStateException: Agent is not connected after 100 seconds, status: Running
    at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:187)
    at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
    at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Jun 17, 2019 2:51:23 PM FINER org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher

Removing Jenkins node: slave-738cb

Jun 17, 2019 2:51:23 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate

Terminating Kubernetes instance for agent slave-738cb

Note: Same setup works perfectly on Google Cloud without any issues.

-- Nitin

jenkins

jenkins-pipeline

kubernetes

1 Answer

3/4/2020

Maybe it's not to answer your question. I had a simular problem when I try to spin up a slave (Kubernetes pod). See below job output

[Pipeline] node
Still waiting to schedule task
Waiting for next available executor
Created Pod: jenkins-pipeline-4ea762be-3222-41f8-a091-620bdca2ad7b-hx2-vd8j7 in namespace tools
Created Pod: jenkins-pipeline-4ea762be-3222-41f8-a091-620bdca2ad7b-hx2-lv33g in namespace tools
Created Pod: jenkins-pipeline-4ea762be-3222-41f8-a091-620bdca2ad7b-hx2-646dm in namespace tools
Ready to run at Wed Mar 04 18:20:20 UTC 2020
Resuming build at Wed Mar 04 18:20:20 UTC 2020 after Jenkins restart

You can see Jenkins says it has been created but when I check the pod it was keep re-creating.

Jenkins logs

2020-03-04 18:27:07.295+0000 [id=220]   WARNING o.c.j.p.k.KubernetesLauncher#launch: Error in provisioning; agent=KubernetesSlave name: jenkins-pipeline-4ea762be-3222-41f8-a091-620bdca2ad7b-hx2-t14q4, template=PodTemplate{, name='jenkins-pipeline-4ea762be-3222-41f8-a091-620bdca2ad7b-hx2z3', namespace='tools', label='jenkins-pipeline-4ea762be-3222-41f8-a091-620bdca2ad7b', nodeUsageMode=EXCLUSIVE, workspaceVolume=EmptyDirWorkspaceVolume [memory=false], annotations=[org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@aab9c821, org.csanchez.jenkins.plugins.kubernetes.PodAnnotation@c92c82e4]}
io.fabric8.kubernetes.client.KubernetesClientTimeoutException: Timed out waiting for [100000] milliseconds for [Pod] with name:[jenkins-pipeline-4ea762be-3222-41f8-a091-620bdca2ad7b-hx2-t14q4] in namespace [tools].
    at org.csanchez.jenkins.plugins.kubernetes.AllContainersRunningPodWatcher.await(AllContainersRunningPodWatcher.java:147)
    at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:139)
    at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:292)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
    at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
2020-03-04 18:27:07.295+0000 [id=220]   INFO    o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent jenkins-pipeline-4ea762be-3222-41f8-a091-620bdca2ad7b-hx2-t14q4
2020-03-04 18:27:07.312+0000 [id=220]   INFO    o.c.j.p.k.KubernetesSlave#deleteSlavePod: Terminated Kubernetes instance for agent tools/jenkins-pipeline-4ea762be-3222-41f8-a091-620bdca2ad7b-hx2-t14q4
Terminated Kubernetes instance for agent tools/jenkins-pipeline-4ea762be-3222-41f8-a091-620bdca2ad7b-hx2-t14q4
2020-03-04 18:27:07.312+0000 [id=220]   INFO    o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer jenkins-pipeline-4ea762be-3222-41f8-a091-620bdca2ad7b-hx2-t14q4

It turns out my slave was looking for a secret

  Warning  FailedMount  4s (x4 over 7s)  kubelet, gke-fuchicorp-cluster-default-pool-f07211fa-06b8  MountVolume.SetUp failed for volume "google-service-account" : secret "fuchicorp-service-account" not found

You can see that secret does not exist in that namespace

kubectl get secret -n tools fuchicorp-service-account
Error from server (NotFound): secrets "fuchicorp-service-account" not found

I just re-created that needed secret then pod was scheduled and

kubectl get pod -n tools | grep jenkins
jenkins-master-tools-57946f6b9-v449m                          1/1     Running   0          35h
jenkins-pipeline-4ea762be-3222-41f8-a091-620bdca2ad7b-hx2-pvdgt   3/3     Running   0          33s

After that, my job was successfully build

Resuming build at Wed Mar 04 18:20:20 UTC 2020 after Jenkins restart
Agent jenkins-pipeline-4ea762be-3222-41f8-a091-620bdca2ad7b-hx2-pvdgt is provisioned from template jenkins-pipeline-4ea762be-3222-41f8-a091-620bdca2ad7b-hx2z3
---
apiVersion: "v1"
kind: "Pod"
metadata:
  annotations:
    jenkinsjoblabel: "academy-fuchicorp-build/dev-3"
    buildUrl: "http://jenkins-deployment-tools:443/job/academy-fuchicorp-build/job/dev/3/"
    runUrl: "job/academy-fuchicorp-build/job/dev/3/"
  labels:
    k8s-label: "jenkins-pipeline-4ea762be-3222-41f8-a091-620bdca2ad7b"
    jenkins: "slave"
    jenkins/label: "jenkins-pipeline-4ea762be-3222-41f8-a091-620bdca2ad7b"

Suggestions:

Make sure all required manifests are created for you slave
Describe the pod (slave) and you will see some outputs.
Check that master can talk to Kubernetes cluster by running kubectl get node inside master pod

-- Farkhod Sadykov

Source: StackOverflow

KQ

Jenkins jnlp slave not working on local Kubernetes cluster

Similar Questions

1 Answer

K
Q