Jenkins setup:
Jenkins: 2.467
OS: Linux - 5.4.0-177-generic
Java: 17.0.11 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
Parameterized-Remote-Trigger:3.2.0
ace-editor:1.1
active-directory:2.35
ansicolor:1.0.4
ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
apache-httpcomponents-client-5-api:5.3.1-110.v77252fb_d4da_5
artifactory:4.0.8
asm-api:9.7-33.v4d23ef79fcc8
authentication-tokens:1.119.v50285141b_7e1
badge:1.13
blueocean:1.27.13
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.13
blueocean-commons:1.27.13
blueocean-config:1.27.13
blueocean-core-js:1.27.13
blueocean-dashboard:1.27.13
blueocean-display-url:2.4.3
blueocean-events:1.27.13
blueocean-git-pipeline:1.27.13
blueocean-github-pipeline:1.27.13
blueocean-i18n:1.27.13
blueocean-jwt:1.27.13
blueocean-personalization:1.27.13
blueocean-pipeline-api-impl:1.27.13
blueocean-pipeline-editor:1.27.13
blueocean-pipeline-scm-api:1.27.13
blueocean-rest:1.27.13
blueocean-rest-impl:1.27.13
blueocean-web:1.27.13
bootstrap4-api:4.6.0-6
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.78.1-248.ve27176eb_46cb_
branch-api:2.1169.va_f810c56e895
build-user-vars-plugin:166.v52976843b_435
build-with-parameters:76.v9382db_f78962
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.2.0
cloud-stats:336.v788e4055508b_
cloudbees-bitbucket-branch-source:888.v8e6d479a_1730
cloudbees-folder:6.942.vb_43318a_156b_2
command-launcher:107.v773860566e2e
commons-compress-api:1.26.1-2
commons-lang3-api:3.14.0-76.vda_5591261cfe
commons-text-api:1.12.0-119.v73ef73f2345d
conditional-buildstep:1.4.3
config-file-provider:973.vb_a_80ecb_9a_4d0
configuration-as-code:1810.v9b_c30a_249a_4c
credentials:1371.vfee6b_095f0a_3
credentials-binding:681.vf91669a_32e45
data-tables-api:2.0.8-1
display-url-api:2.204.vf6fddd8a_8b_e9
docker-commons:439.va_3cb_0a_6a_fb_29
docker-java-api:3.3.6-90.ve7c5c7535ddd
docker-plugin:1.6.2
durable-task:555.v6802fe0f0b_82
echarts-api:5.5.0-1
eddsa-api:0.3.0-4.v84c6f0f4969e
email-ext:1814.v404722f34263
emailext-template:1.5
envinject:2.908.v66a_774b_31d93
envinject-api:1.199.v3ce31253ed13
external-monitor-job:215.v2e88e894db_f8
favorite:2.218.vd60382506538
font-awesome-api:6.5.2-1
gerrit-trigger:2.41.1
git:5.2.2
git-client:5.0.0
git-parameter:0.9.19
git-server:99.va_0826a_b_cdfa_d
github:1.39.0
github-api:1.318-461.v7a_c09c9fa_d63
github-branch-source:1789.v5b_0c0cea_18c3
gitlab-api:5.3.0-91.v1f9a_fda_d654f
gitlab-plugin:1.8.1
gradle:2.12
gson-api:2.11.0-41.v019fcf6125dc
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
hashicorp-vault-plugin:368.v48134f694db_f
hidden-parameter:237.v4b_df26c7a_f0e
htmlpublisher:1.36
hudson-wsclean-plugin:1.0.8
instance-identity:185.v303dc7c645f9
ionicons-api:74.v93d5eb_813d5f
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javadoc:243.vb_b_503b_b_45537
javax-activation-api:1.2.0-7
javax-mail-api:1.6.2-10
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jenkins-design-language:1.27.13
jersey2-api:2.42-147.va_28a_44603b_d5
jira:3.13
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
job-dsl:1.87
job-restrictions:0.8
jobConfigHistory:1229.v3039470161a_d
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery:1.12.4-1
jquery3-api:3.7.1-2
jsch:0.2.16-86.v42e010d9484b_
json-api:20240303-41.v94e11e6de726
json-path-api:2.9.0-58.v62e3e85b_a_655
junit:1265.v65b_14fa_f12f0
kubernetes:4253.v7700d91739e5
kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
kubernetes-credentials:174.va_36e093562d9
ldap:725.v3cb_b_711b_1a_ef
leastload:3.0.0
locale:519.v4e20f313cfa_f
lockable-resources:1255.vf48745da_35d0
mailer:472.vf7c289a_4b_420
mask-passwords:173.v6a_077a_291eb_5
matrix-auth:3.2.2
matrix-project:832.va_66e270d2946
maven-plugin:3.23
metrics:4.2.21-451.vd51df8df52ec
mina-sshd-api-common:2.13.1-117.v2f1a_b_66ff91d
mina-sshd-api-core:2.13.1-117.v2f1a_b_66ff91d
momentjs:1.1.1
monitoring:1.99.0
notification:1.17
oic-auth:4.297.vcddb_d8a_e4694
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
pam-auth:1.11
parameterized-scheduler:277.v61a_4b_a_49a_c5c
parameterized-trigger:806.vf6fff3e28c3e
permissive-script-security:0.7
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-groovy-lib:727.ve832a_9244dfa_
pipeline-input-step:495.ve9c153f6067b_
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2205.vc9522a_9d5711
pipeline-model-definition:2.2205.vc9522a_9d5711
pipeline-model-extensions:2.2205.vc9522a_9d5711
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2205.vc9522a_9d5711
pipeline-stage-view:2.34
pipeline-utility-steps:2.17.0
plain-credentials:183.va_de8f1dd5a_2b_
plugin-util-api:4.1.0
popper-api:1.16.1-3
popper2-api:2.11.6-5
prism-api:1.29.0-15
pubsub-light:1.18
rebuild:332.va_1ee476d8f6d
resource-disposer:0.23
role-strategy:727.vd344b_eec783d
run-condition:1.7
saml:4.464.vea_cb_75d7f5e0
scm-api:690.vfc8b_54395023
script-security:1341.va_2819b_414686
shared-objects:0.44
simple-theme-plugin:191.vcd207ef9dd24
skip-certificate-check:1.1
snakeyaml-api:2.2-111.vc6598e30cc65
sse-gateway:1.27
ssh-credentials:337.v395d2403ccd4
ssh-slaves:2.973.v0fa_8c0dea_f9f
sshd:3.330.vc866a_8389b_58
structs:338.v848422169819
thinBackup:2.1.1
throttle-concurrents:2.14
timestamper:1.27
token-macro:400.v35420b_922dcb_
trilead-api:2.147.vb_73cc728a_32e
uno-choice:2.8.3
variant:60.v7290fc0eb_b_cd
windows-slaves:1.8.1
workflow-aggregator:600.vb_57cdd26fdd7
workflow-api:1316.v33eb_726c50b_a_
workflow-basic-steps:1058.vcb_fc1e3a_21a_9
workflow-cps:3908.vd6b_b_5a_a_54010
workflow-durable-task-step:1360.v82d13453da_a_f
workflow-job:1436.vfa_244484591f
workflow-multibranch:791.v28fb_f74dfca_e
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:678.v3ee58b_469476
workflow-support:920.v59f71ce16f04
ws-cleanup:0.46
Quick note: we had encountered this issue in our production cluster which has lower version of Jenkins core and plugins, but I’m able to replicate it in a controlled environment with Jenkins set to latest.
We have a primary build pipeline that uses workflow-cps parallel step to run around ~50 containers at the same time, and it’s heavily sh dependent.
As more agents are added to the parallel step, sh commands take longer to complete if they are wrapped in groovy script.
I’ve simplified the pipeline so it will be easier to reproduce:
pipeline {
agent none
stages {
stage('Build and Test') {
parallel {
stage('Stage1') {
agent {
kubernetes {
yaml """
apiVersion: v1
kind: Pod
metadata:
name: test-pod
spec:
containers:
- name: test-build
image: ubuntu
resources:
requests:
memory: 4000Mi
cpu: 4
limits:
memory: 4000Mi
cpu: 4
command: ['sleep']
args: ['6h']
tty: true
"""
}
}
steps {
container('test-build') {
script{
for (int i = 0; i < 30; i++) {
sh 'cat /etc/hosts'
}
}
}
}
}
stage('Stage2') {
agent {
kubernetes {
yaml """
apiVersion: v1
kind: Pod
metadata:
name: test-pod
spec:
containers:
- name: test-build
image: ubuntu
resources:
requests:
memory: 4000Mi
cpu: 4
limits:
memory: 4000Mi
cpu: 4
command: ['sleep']
args: ['6h']
tty: true
"""
}
}
steps {
container('test-build') {
script{
for (int i = 0; i < 30; i++) {
sh 'cat /etc/hosts'
}
}
}
}
}
// ...
// ....
// in here you can keep pasting duplicates of stages
}
}
}
}
Apologizes for hard coding each stage, but I wasn’t able to come up with a loop that does it better.
My results are as follows:
7 stages: ~50 seconds for each agent to finish
17 stages: ~120 seconds for each agent to finish
35 stages: ~270 for each agent to finish
If I leave only one stage at the parallel part, then run the same build X50 times at the same time, each build finishes very fast (around 15 seconds)
So I don’t think it’s a general load issue, but rather maybe parallel step is throttling the sh response in some way, making them hang. (maybe CpsFlowExecution thread?)
Would apricate your advise.