Jenkins forces to increase Max connections to Kubernetes API

1/8/2019
Jenkins version : 2.121.3

I am using the k8s plugin in Jenkins which helps me deploy my sequential and parallel jobs on my k8s cluster.

Here is the part of the Jenkinsfile where the job fails

           parallel([
                    build: {
                        stage('check formatting')
                        // some code

                        stage('build')
                        // some code

                        stage('build image')
                        // some code

                        stage('push image') 
                        // some code
                    },
                    test: {
                        stage('test') 
                        // some code

                        stage('build e2e test image')
                        // some code

                        }

                        stage('push e2e test image') 
                        // some code
                    },
                   failFast: true])

While running a job which has parallel stages, I come across the error

caught java.io.IOException: Connection was rejected, you should increase the Max connections to Kubernetes API

Then I goto Manage Jenkins --> Configure System --> Cloud --> Kubernetes and increase "Max connections to Kubernetes API" by say.... 20 and re-run the job and it works. And if I try to re-run the job after that, it fails with the same error and then again I have to increase the limit. So Im kind of in an endless loop doing the following:

1) Restart jenkins evrerytime

2) Bump up API Server requests limit.

So Im not sure why this is happening. Is there a way for me to find out how many requests are being handled by the API server ? And whats their source ? And how can I find out if there are any lingering requests from previous jobs ?

Full error log :

Failed in branch build
[Pipeline] // parallel
[Pipeline] echo
caught java.io.IOException: Connection was rejected, you should increase the Max connections to Kubernetes API
[Pipeline] echo
org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:329)
org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:237)
hudson.Launcher$ProcStarter.start(Launcher.java:449)
org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:188)
org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:99)
org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:278)
org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:270)
org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:178)
org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
sun.reflect.GeneratedMethodAccessor646.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:157)
org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:155)
org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:159)
org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
WorkflowScript.run(WorkflowScript:109)
___cps.transform___(Native Method)
com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
sun.reflect.GeneratedMethodAccessor243.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:103)
com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
sun.reflect.GeneratedMethodAccessor243.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:60)
com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
sun.reflect.GeneratedMethodAccessor243.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
com.cloudbees.groovy.cps.Next.step(Next.java:83)
com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)
org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)
org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)
org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:182)
org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:836)
--
jenkins
kubernetes
kubernetes-apiserver

2 Answers

1/11/2019

Set the Max connections to Kubernetes API to 64. Sometimes, setting it too low will generate errors. Also, Set the Container Cleanup Timeout to 5.

-- Mike
Source: StackOverflow

1/14/2019

it's looks it is the official bug of plugin

https://issues.jenkins-ci.org/browse/JENKINS-55392

You can try comment this issue, to help community to solve the problem.

-- Tirex
Source: StackOverflow