K
Q

Parallel jenkins agents at kubernetes with kubernetes plugin

10/31/2019

I'm using Jenkins version 2.190.2 and Kubernetes plugin 1.19.0 I have this jenkins as master at kubernetes cluster at AWS. This jenkins have kubernetes plugin configured and it's running ok. I have some pod templates and containers configured that are running. I'm able to run declarative pipelines specifying agent and container.

My problem is that I'm unable to run jobs parallel. When more than one job is executed at same time, first job starts, pod is created and execute stuff. The second job waits to the first jobs ends, even if use different agents.

EXAMPLE:

Pipeline 1

pipeline {
    agent { label "bash" }

    stages {
        stage('init') {
            steps {
                container('bash') {
                    echo 'bash'
                    sleep 300
                }
            }
        }
    }
}

Pipeline 2

pipeline {
    agent { label "bash2" }

    stages {
        stage('init') {
            steps {
                container('bash2') {
                    echo 'bash2'
                    sleep 300
                }
            }
        }
    }
}

This is the org.csanchez.jenkins.plugins.kubernetes log. I've uploaded to wetransfer -> we.tl/t-ZiSbftKZrK

I've read a lot of this problem and I've configured jenkins start with this JAVA_OPTS but problem is not solved.

-Dhudson.slaves.NodeProvisioner.initialDelay=0 
-Dhudson.slaves.NodeProvisioner.MARGIN=50 
-Dhudson.slaves.NodeProvisioner.MARGIN0=0.85

Kubernetes plugin is configured with:

  • Kubernetes cloud / Concurrency Limit = 50. I've configured without value but the problem still occurs
  • Kubernetes cloud / Pod retention = never
  • Pod template / Concurrency Limit without value. I've configured with 10 but the problem still occurs
  • Pod template / Pod retention = Default

What configuration I'm missing or what errors i'm doing?

-- jagr
jenkins
jenkins-pipeline
jenkins-plugins
kubernetes

Similar Questions

Kubernetes nginx ingress path-based routing of HTTPS in AWS
helm test via Jenkins pipeline
Ephemeral Storage Garbage Collection
K8s Cluster over Azure using Kubeadm + Autoscaling using VMSS
Why isn't the circuit breaking of ISTIO working?

3 Answers

11/15/2019

Finally I've solved my problem due to another problem. We started to get errors at create normal pods because our kubernetes node at aws hadn't enough free ip's. Due to this error we scaled our nodes and now jenkins pipelines can be running parallel with diferents pods and containers.

-- jagr
Source: StackOverflow
11/1/2019

your pods are created in parallel

Oct 31, 2019 3:13:30 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
Created Pod: default/bash-4wjrk
...
Oct 31, 2019 3:13:30 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
Created Pod: default/bash2-3rxck

but your bash2 pod is failing with

Caused by: java.net.UnknownHostException: jenkins-jnlp.default.svc.cluster.local
-- csanchez
Source: StackOverflow
11/1/2019

You should use Parallel Stages. Which you can find described in the Jenkins documentation for pipeline syntax.

Stages in Declarative Pipeline may declare a number of nested stages within a parallel block, which will be executed in parallel. Note that a stage must have one and only one of steps, stages, or parallel. The nested stages cannot contain further parallel stages themselves, but otherwise behave the same as any other stage, including a list of sequential stages within stages. Any stage containing parallel cannot contain agent or tools, since those are not relevant without steps.

In addition, you can force your parallel stages to all be aborted when one of them fails, by adding failFast true to the stage containing the parallel. Another option for adding failfast is adding an option to the pipeline definition: parallelsAlwaysFailFast()

An example pipeline might look like this:

Jenkinsfile
pipeline {
    agent none
    stages {
        stage('Run pod') {
            parallel {
                stage('bash') {
                    agent {
                        label "init"
                    }
                    steps {
                        container('bash') {
                            echo 'bash'
                            sleep 300
                        }
                    }
                }
                stage('bash2') {
                    agent {
                        label "init"
                    }
                    steps {
                        container('bash') {
                            echo 'bash'
                            sleep 300
                        }
                    }
                }
            }
        }
    }
}
-- Crou
Source: StackOverflow