Jenkins version : 2.121.3
I am using the k8s plugin in Jenkins which helps me deploy my sequential and parallel jobs on my k8s cluster.
Here is the part of the Jenkinsfile
where the job fails
parallel([
build: {
stage('check formatting')
// some code
stage('build')
// some code
stage('build image')
// some code
stage('push image')
// some code
},
test: {
stage('test')
// some code
stage('build e2e test image')
// some code
}
stage('push e2e test image')
// some code
},
failFast: true])
While running a job which has parallel stages, I come across the error
caught java.io.IOException: Connection was rejected, you should increase the Max connections to Kubernetes API
Then I goto Manage Jenkins --> Configure System --> Cloud --> Kubernetes and increase "Max connections to Kubernetes API" by say.... 20 and re-run the job and it works. And if I try to re-run the job after that, it fails with the same error and then again I have to increase the limit. So Im kind of in an endless loop doing the following:
1) Restart jenkins evrerytime
2) Bump up API Server requests limit.
So Im not sure why this is happening. Is there a way for me to find out how many requests are being handled by the API server ? And whats their source ? And how can I find out if there are any lingering requests from previous jobs ?
Full error log :
Failed in branch build
[Pipeline] // parallel
[Pipeline] echo
caught java.io.IOException: Connection was rejected, you should increase the Max connections to Kubernetes API
[Pipeline] echo
org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:329)
org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:237)
hudson.Launcher$ProcStarter.start(Launcher.java:449)
org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:188)
org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:99)
org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:278)
org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:270)
org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:178)
org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
sun.reflect.GeneratedMethodAccessor646.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:157)
org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:155)
org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:159)
org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
WorkflowScript.run(WorkflowScript:109)
___cps.transform___(Native Method)
com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
sun.reflect.GeneratedMethodAccessor243.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:103)
com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
sun.reflect.GeneratedMethodAccessor243.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:60)
com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
sun.reflect.GeneratedMethodAccessor243.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
com.cloudbees.groovy.cps.Next.step(Next.java:83)
com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)
org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)
org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)
org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:182)
org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:836)
Set the Max connections to Kubernetes API to 64. Sometimes, setting it too low will generate errors. Also, Set the Container Cleanup Timeout to 5.
it's looks it is the official bug of plugin
https://issues.jenkins-ci.org/browse/JENKINS-55392
You can try comment this issue, to help community to solve the problem.