Jenkins timeout waiting for container to become ready

1/23/2017

I'm running Jenkins on OpenShift Origin and Fabric8.

OpenShift Master: v1.3.2 Kubernetes Master: v1.3.0+52492b4

I'm having boat loads of problems with jenkins and farbic8!

One of the latest issues is an error - happens everytime - when trying to build in Jenkins.

Anyone know how to fix this?

The error is:

Waiting for container container [maven] of pod [jenkins-1-1d0ts] to become ready.
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
java.io.IOException: Failed to execute shell script inside container [maven] of pod [jenkins-1-1d0ts]. Timed out waiting for container to become ready!
    at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:88)
    at hudson.Launcher$ProcStarter.start(Launcher.java:384)
    at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:147)
    at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:61)
    at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:172)
    at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:184)
    at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:126)
    at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
    at groovy.lang.GroovyObject$invokeMethod.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:151)
    at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:21)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:115)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:103)
    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:149)
    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:146)
    at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:16)
    at mavenCanaryRelease.call(/var/jenkins_home/jobs/shiftwork-staffservice-pipeline-dev/builds/6/libs/github.com/fabric8io/fabric8-pipeline-library/vars/mavenCanaryRelease.groovy:11)
    at WorkflowScript.run(WorkflowScript:39)
    at ___cps.transform___(Native Method)
    at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
    at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:103)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
    at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
    at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:60)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
    at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
    at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
    at com.cloudbees.groovy.cps.Next.step(Next.java:58)
    at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
    at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:163)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:324)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:78)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:236)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:224)
    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:63)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Finished: FAILURE

I also noticed the following. There is nothing unusual going on with the pods. But it seems Jenkins cannot connect to Kubenetes. enter image description here

All pods appear to be running correctly.

When I retrieve the logs from the Jenkins pod I see:

INFO: Finished Download metadata. 6,467 ms
--> setting agent port for jnlp
--> setting agent port for jnlp... done
Jan 23, 2017 9:11:08 PM hudson.model.AsyncPeriodicWork$1 run
INFO: Started Gravatar periodic lookup
Jan 23, 2017 9:11:08 PM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished Gravatar periodic lookup. 1 ms
Jan 23, 2017 9:11:23 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: shiftwork-staffservice-pipeline-dev #21 completed: FAILURE
Jan 23, 2017 9:11:23 PM io.jenkins.blueocean.events.PipelineEventListener$1 run
SEVERE: Unexpected error publishing pipeline FlowNode event.
java.util.concurrent.ExecutionException: hudson.AbortException
    at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:289)
    at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:276)
    at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
    at io.jenkins.blueocean.events.PipelineEventListener$1.run(PipelineEventListener.java:226)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: hudson.AbortException
    at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1068)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1094)
    at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109)
    at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:108)
    at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:57)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:215)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:404)
-- Magick
fabric8
jenkins
kubernetes
openshift

1 Answer

1/28/2017

This looks like the same Jenkins issue I reported here https://issues.jenkins-ci.org/browse/JENKINS-41388

TL;DR If Jenkins is running in a namespace other than 'default' then any change to jenkins configuration incorrectly also updates the namespace to run build pods. Until this is fixed in Jenkins a workaround is to manually change that 'Kubernetes Namespace' field to the same namespace as Jenkins is running. This should avoid the issue with the service account.

-- James Rawlings
Source: StackOverflow