Hello everyone,
We have a Jenkins job that distributes nunit tests across 400 VMs. Recently, whenever the pipeline reaches the stage where some steps get executed on the VMs parallel, we have started seeing the following error:
Any idea what could cause this error?
Jenkins setup:
Jenkins: 2.440.1
OS: Windows Server 2019 - 10.0
Java: 17.0.3 - Microsoft (OpenJDK 64-Bit Server VM)
active-directory:2.35
ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
authentication-tokens:1.53.v1c90fd9191a_b_
authorize-project:1.7.1
blueocean:1.27.11
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.11
blueocean-commons:1.27.11
blueocean-config:1.27.11
blueocean-core-js:1.27.11
blueocean-dashboard:1.27.11
blueocean-display-url:2.4.2
blueocean-events:1.27.11
blueocean-git-pipeline:1.27.11
blueocean-github-pipeline:1.27.11
blueocean-i18n:1.27.11
blueocean-jira:1.27.11
blueocean-jwt:1.27.11
blueocean-personalization:1.27.11
blueocean-pipeline-api-impl:1.27.11
blueocean-pipeline-editor:1.27.11
blueocean-pipeline-scm-api:1.27.11
blueocean-rest:1.27.11
blueocean-rest-impl:1.27.11
blueocean-web:1.27.11
bootstrap5-api:5.3.2-4
bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9
branch-api:2.1152.v6f101e97dd77
build-name-setter:2.4.1
build-timeout:1.32
caffeine-api:3.1.8-133.v17b_1ff2e0599
change-assembly-version-plugin:1.10
checks-api:2.0.2
cloudbees-bitbucket-branch-source:876.v857269a_5f439
cloudbees-folder:6.901.vb_4c7a_da_75da_3
command-launcher:107.v773860566e2e
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.11.0-95.v22a_d30ee5d36
conditional-buildstep:1.4.3
copyartifact:722.v0662a_9b_e22a_c
credentials:1319.v7eb_51b_3a_c97b_
credentials-binding:657.v2b_19db_7d6e6d
description-setter:239.vd0a_6b_785f92d
display-url-api:2.200.vb_9327d658781
docker-commons:439.va_3cb_0a_6a_fb_29
docker-workflow:572.v950f58993843
durable-task:550.v0930093c4b_a_6
echarts-api:5.4.3-4
elastic-axis:464.va_7ed499b_9d75
email-ext:2.104
envinject:2.908.v66a_774b_31d93
envinject-api:1.199.v3ce31253ed13
external-monitor-job:215.v2e88e894db_f8
external-workspace-manager:1.3.1
favorite:2.208.v91d65b_7792a_c
font-awesome-api:6.5.1-3
git:5.2.1
git-client:4.6.0
git-server:114.v068a_c7cc2574
github:1.38.0
github-api:1.318-461.v7a_c09c9fa_d63
github-branch-source:1772.va_69eda_d018d4
groovy:457.v99900cb_85593
gson-api:2.10.1-15.v0d99f670e0a_7
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
htmlpublisher:1.32
instance-identity:185.v303dc7c645f9
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.16.1-373.ve709c6871598
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javadoc:243.vb_b_503b_b_45537
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jenkins-design-language:1.27.11
jersey2-api:2.41-133.va_03323b_a_1396
jira:3.12
jjwt-api:0.11.5-77.v646c772fddb_0
job-import-plugin:3.6
job-restrictions:0.8
jobConfigHistory:1229.v3039470161a_d
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery:1.12.4-1
jquery3-api:3.7.1-2
jsch:0.2.16-86.v42e010d9484b_
json-path-api:2.9.0-33.v2527142f2e1d
junit:1259.v65ffcef24a_88
ldap:711.vb_d1a_491714dc
mailer:463.vedf8358e006b_
matrix-auth:3.2.1
matrix-combinations-parameter:1.3.3
matrix-project:822.824.v14451b_c0fd42
maven-plugin:3.23
mercurial:1260.vdfb_723cdcc81
mina-sshd-api-common:2.12.0-90.v9f7fb_9fa_3d3b_
mina-sshd-api-core:2.12.0-90.v9f7fb_9fa_3d3b_
monitoring:1.98.0
msbuild:1.30
nodelabelparameter:1.12.0
nunit:485.ve8a_85357320d
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
p4:1.14.4
pam-auth:1.10
parameterized-trigger:787.v665fcf2a_830b_
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:704.vc58b_8890a_384
pipeline-input-step:491.vb_07d21da_1a_fb_
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2175.v76a_fff0a_2618
pipeline-model-definition:2.2175.v76a_fff0a_2618
pipeline-model-extensions:2.2175.v76a_fff0a_2618
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2175.v76a_fff0a_2618
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:4.1.0
powershell:2.1
pubsub-light:1.18
purge-build-queue-plugin:88.v23b_97b_f2c7a_d
purge-job-history:1.6
resource-disposer:0.23
role-strategy:689.v731678c3e0eb_
run-condition:1.7
saferestart:0.7
scm-api:683.vb_16722fb_b_80b_
script-security:1326.vdb_c154de8669
snakeyaml-api:2.2-111.vc6598e30cc65
sonar:2.17.2
sse-gateway:1.26
ssh-credentials:308.ve4497b_ccd8f4
sshd:3.322.v159e91f6a_550
structs:337.v1b_04ea_4df7c8
swarm:3.44
test-results-analyzer:0.4.1
thinBackup:1.18
timestamper:1.26
token-macro:400.v35420b_922dcb_
trilead-api:2.133.vfb_8a_7b_9c5dd1
variant:60.v7290fc0eb_b_cd
workflow-aggregator:596.v8c21c963d92d
workflow-api:1291.v51fd2a_625da_7
workflow-basic-steps:1042.ve7b_140c4a_e0c
workflow-cps:3867.v535458ce43fd
workflow-durable-task-step:1331.vc8c2fed35334
workflow-job:1400.v7fd111b_ec82f
workflow-multibranch:773.vc4fe1378f1d5
workflow-scm-step:415.v434365564324
workflow-step-api:657.v03b_e8115821b_
workflow-support:865.v43e78cc44e0d
ws-cleanup:0.45