K
Q

Jenkins, MiniKube & Kubernetes Plugin - Dynamically spawning slaves

8/28/2017

Problem

I have deployed csanchez's jenkins-kubernetes plugin (version 0.12) to a local Minikube / Kubernetes environment. When configuring Pod-Templates and Container-Templates from within the Jenkins UI, the PODS are spawned automatically and process simple jobs. Howerver when the Pod-Templates and Container-Templates are defined within a pipeLine script, the Jenkins Master is rejecting the connection saying that the POD is already connected to the master.

Environment

minikube version: v0.20.0

Kubernetes version: v1.7.1

Jenkins Master: jenkins:2.74-alpine (Docker Container)

Jenkins Slave: jnlp-slave:3.10-1-alpine (Docker Container)

Operating System: 4.8.0-59-generic #64-Ubuntu SMP Thu Jun 29 19:38:34 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

PipeLine Script

podTemplate(name: 'my-pod-template', label: 'maven', cloud: 
'kubernetes', namespace: 'mmidevops', containers: [
  containerTemplate(
    name: 'my-container-template', 
    image: 'grantericedwards/jenkinsci-jnlp-slave:3.10-1-alpine', 
    ttyEnabled: true, 
    alwaysPullImage: true,
    workingDir: '/home/jenkins'
  )
  ],
volumes: [
    hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock')
]
) {

  node('maven') {
    stage('Build a Maven project') {
      container('maven') {
          sh 'sleep 10'
      }
    }
  }
}

Spawned PODS kubectl get pods --all-namespaces -w

NAMESPACE     NAME                          READY     STATUS    RESTARTS   AGE
kube-system   kube-addon-manager-minikube   1/1       Running   9          7d
kube-system   kube-dns-1301475494-zp3tg     3/3       Running   27         7d
kube-system   kubernetes-dashboard-jlsk7    1/1       Running   9          7d
mmidevops     jenkins-83791151-msdcw        1/1       Running   30         6d
mmidevops   my-pod-template-nfdgh-9k7qh   0/2       Pending   0         0s
mmidevops   my-pod-template-nfdgh-9k7qh   0/2       Pending   0         0s
mmidevops   my-pod-template-nfdgh-9k7qh   0/2       ContainerCreating   0         0s
mmidevops   my-pod-template-nfdgh-9k7qh   2/2       Running   0         6s
mmidevops   my-pod-template-nfdgh-9k7qh   1/2       Error     0         9s

Logs From Slave POD

INFO: Setting up slave: my-pod-template-nfdgh-9k7qh
Aug 28, 2017 11:35:20 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Aug 28, 2017 11:35:20 AM hudson.remoting.Engine startEngine
WARNING: No Working Directory. Using the legacy JAR Cache location: /home/jenkins/.jenkins/cache/jars
Aug 28, 2017 11:35:20 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://172.17.0.2:8080/]
Aug 28, 2017 11:35:20 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, JNLP-connect, Ping, JNLP2-connect]
Aug 28, 2017 11:35:20 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
  Agent address: 172.17.0.2
  Agent port:    50000
  Identity:      74:24:d0:2e:7b:b7:9d:13:80:47:e5:fa:45:b3:85:15
Aug 28, 2017 11:35:20 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Aug 28, 2017 11:35:20 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to 172.17.0.2:50000
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: 74:24:d0:2e:7b:b7:9d:13:80:47:e5:fa:45:b3:85:15
Aug 28, 2017 11:35:21 AM org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer onRecv
INFO: [JNLP4-connect connection to 172.17.0.2/172.17.0.2:50000] Local headers refused by remote: my-pod-template-nfdgh-9k7qh is already connected to this master. Rejecting this connection.
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: my-pod-template-nfdgh-9k7qh is already connected to this master. Rejecting this connection.
at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
at hudson.remoting.Engine.innerRun(Engine.java:583)
at hudson.remoting.Engine.run(Engine.java:447)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: my-pod-template-nfdgh-9k7qh is already connected to this master. Rejecting this connection.
at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.newAbortCause(ConnectionHeadersFilterLayer.java:377)
at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.onRecvClosed(ConnectionHeadersFilterLayer.java:432)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:48)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:247)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at hudson.remoting.Engine$1$1.run(Engine.java:98)
at java.lang.Thread.run(Thread.java:748)
Suppressed: java.nio.channels.ClosedChannelException
    ... 7 more

Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to 172.17.0.2:50000
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Server reports protocol JNLP4-plaintext not supported, skipping
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP3-connect is not enabled, skipping
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP2-connect
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP2-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at hudson.remoting.Engine.innerRun(Engine.java:583)
at hudson.remoting.Engine.run(Engine.java:447)

Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
at org.jenkinsci.remoting.engine.JnlpProtocol2Handler.sendHandshake(JnlpProtocol2Handler.java:134)
at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:162)
at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:158)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at hudson.remoting.Engine$1$1.run(Engine.java:98)
at java.lang.Thread.run(Thread.java:748)

Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to 172.17.0.2:50000
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP-connect
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at hudson.remoting.Engine.innerRun(Engine.java:583)
at hudson.remoting.Engine.run(Engine.java:447)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
at org.jenkinsci.remoting.engine.JnlpProtocol1Handler.sendHandshake(JnlpProtocol1Handler.java:121)
at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:162)
at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:158)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at hudson.remoting.Engine$1$1.run(Engine.java:98)
at java.lang.Thread.run(Thread.java:748)

Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: The server rejected the connection: None of the protocols were accepted
java.lang.Exception: The server rejected the connection: None of the protocols were accepted
at hudson.remoting.Engine.onConnectionRejected(Engine.java:644)
at hudson.remoting.Engine.innerRun(Engine.java:608)
at hudson.remoting.Engine.run(Engine.java:447)

Logs From Jenkins Master

Aug 28, 2017 11:35:14 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template: Kubernetes Pod Template
Aug 28, 2017 11:35:14 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
Aug 28, 2017 11:35:14 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
INFO: Created Pod: my-pod-template-nfdgh-9k7qh
Aug 28, 2017 11:35:14 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
INFO: Waiting for Pod to be scheduled (0/100): my-pod-template-nfdgh-9k7qh
Aug 28, 2017 11:35:15 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #38 from /172.17.0.5:54934
Aug 28, 2017 11:35:21 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #39 from /172.17.0.5:54944
Aug 28, 2017 11:35:21 AM org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer onRecv
INFO: [JNLP4-connect connection from 172.17.0.5/172.17.0.5:54944] Refusing headers from remote: my-pod-template-nfdgh-9k7qh is already connected to this master. Rejecting this connection.
Aug 28, 2017 11:35:21 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP2-connect connection #40 from /172.17.0.5:54960
Aug 28, 2017 11:35:21 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP-connect connection #41 from /172.17.0.5:54962
Aug 28, 2017 11:35:23 AM hudson.node_monitors.ResponseTimeMonitor$1 monitor
WARNING: Making my-pod-template-qdgzl-29t7l offline because it’s not responding
Aug 28, 2017 11:35:24 AM hudson.slaves.NodeProvisioner$2 run
INFO: Kubernetes Pod Template provisioning successfully completed. We have now 7 computer(s)
Aug 28, 2017 11:40:20 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminating Kubernetes instance for slave my-pod-template-nfdgh-9k7qh
Aug 28, 2017 11:40:20 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminated Kubernetes instance for slave my-pod-template-nfdgh-9k7qh
Aug 28, 2017 11:40:20 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Disconnected computer my-pod-template-nfdgh-9k7qh
Terminated Kubernetes instance for slave my-pod-template-nfdgh-9k7qh
Aug 28, 2017 11:40:20 AM jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
WARNING: Computer.threadPoolForRemoting [#41] for my-pod-template-nfdgh-9k7qh terminated
java.nio.channels.ClosedChannelException
    at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:208)
    at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
    at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
    at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
    at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800)
    at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
    at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:311)
    at hudson.remoting.Channel.close(Channel.java:1304)
    at hudson.remoting.Channel.close(Channel.java:1272)
    at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:708)
    at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:96)
    at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:626)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Aug 28, 2017 11:40:20 AM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: TestPipeLine #94 completed: FAILURE

JNLP Logs

Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior
Warning: SECRET is defined twice in command-line arguments and the environment variable
Warning: AGENT_NAME is defined twice in command-line arguments and the environment variable
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main createEngine
INFO: Setting up slave: my-pod-template-nfdgh-9k7qh
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Aug 28, 2017 11:35:15 AM hudson.remoting.Engine startEngine
WARNING: No Working Directory. Using the legacy JAR Cache location: /home/jenkins/.jenkins/cache/jars
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://172.17.0.2:8080/]
Aug 28, 2017 11:35:15 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, JNLP-connect, Ping, JNLP2-connect]
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
  Agent address: 172.17.0.2
  Agent port:    50000
  Identity:      74:24:d0:2e:7b:b7:9d:13:80:47:e5:fa:45:b3:85:15
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to 172.17.0.2:50000
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: 74:24:d0:2e:7b:b7:9d:13:80:47:e5:fa:45:b3:85:15
Aug 28, 2017 11:35:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected

Update So after a bit more Googling I found this ... "if you have defined a JNLP container in your Pod definition, you need to remove it or rename it to jnlp, otherwise a new container called jnlp will be created" ... that said ...

podTemplate(name: 'default-java-slave', label: 'maven-builder', cloud: 
'kubernetes', namespace: 'mmidevops', containers: [
  containerTemplate(
    name: 'jnlp', 
    image: 'grantericedwards/jenkinsci-jnlp-slave:3.10-1-alpine', 
    ttyEnabled: true, 
    alwaysPullImage: true,
    workingDir: '/home/jenkins'
  )
  ],
volumes: [
    hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: 
'/var/run/docker.sock')
]
) {

  node('maven-builder') {
    stage('Build a Maven project') {
      container('jnlp') {
          sh 'sleep 10'
      }
    }
  }
}
-- Grant Edwards
jenkins
jnlp
kubernetes
minikube

Similar Questions

Istio bookinfo sample deployment The connection has timed out
How does container's exit code take effect on the kubernetes cluster
Run .net core 1.1 test in kubernetes cluster and connect to google compute engine VM's SQL Server
Is it possible to replicate Kubernetes container network environment with docker compose?

1 Answer

9/8/2017

Im using a kubernetes plugin 0.12 and have a problem ClosedChannelException (I think this is not problem on connection to master but a communicate doesn't acceptable then connection timeout and closed) on Jenkins slave too and I solved this problem when I upgrade Jenkins version from 2.60-alpine to 2.75-apline. Jenkins slave working perfectly now. Hopefully this will help you to solve a problem.

PS. Kubernetes plugin has release new version on 1.0 and I also upgraded to new version it working perfectly too!

-- Tat Sang
Source: StackOverflow