We are running Jenkins on google cloud with agents launched using GCE. Versions of all the plugins are (I think) reasonably up to date. Some of the agents are running on spot instances.
When agent machine is terminated sometimes the build will hang. Usually (but not sure it’s related to) on a very long running ant testing job executed using sh. Typically build would look like this:
matrix {
...
stages {
stage('JUnit run') {
steps {
script {
withCredentials(...) {
dir(...) {
sh '''
ant ...
'''
...
The thread dump captured by Jenkins is:
Thread #10
at DSL.parallel(Native Method)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(ModelInterpreter.groovy:266)
...
Thread #138
at DSL.sh(waiting to reconnect to /var/jenkins/workspace/... on lab--centos-7--x64-...; recurrence period: 15000ms; check task scheduled; cancelled? false done? false)
at WorkflowScript.run(WorkflowScript:71)
at DSL.dir(Native Method)
at WorkflowScript.run(WorkflowScript:70)
at DSL.withCredentials(not currently scheduled, or running blocks)
at WorkflowScript.run(WorkflowScript:69)
at DSL.script(Native Method)
at WorkflowScript.run(WorkflowScript:60)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.delegateAndExecute(ModelInterpreter.groovy:137)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.executeSingleStage(ModelInterpreter.groovy:666)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.catchRequiredContextForNode(ModelInterpreter.groovy:395)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.catchRequiredContextForNode(ModelInterpreter.groovy:393)
I have enabled remote debug agent on the Jenkins JVM and when this happens I’ll try to get more information and post it here.
Pointers to code, issue that I might have overlooked etc. will be greatly appreciated.
Jenkins setup:
Jenkins: 2.414.3
OS: Linux - 5.15.133+
Java: 17.0.8.1 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
apache-httpcomponents-client-5-api:5.3.1-1.0
authentication-tokens:1.53.v1c90fd9191a_b_
bootstrap5-api:5.3.2-3
bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9
branch-api:2.1128.v717130d4f816
caffeine-api:3.1.8-133.v17b_1ff2e0599
categorized-view:1.13
checks-api:2.0.2
cloud-stats:336.v788e4055508b_
cloudbees-folder:6.858.v898218f3609d
command-launcher:107.v773860566e2e
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.11.0-95.v22a_d30ee5d36
configuration-as-code:1752.v0ccc3b_6064b_0
copyartifact:722.v0662a_9b_e22a_c
credentials:1319.v7eb_51b_3a_c97b_
credentials-binding:642.v737c34dea_6c2
dashboard-view:2.495.v07e81500c3f2
display-url-api:2.200.vb_9327d658781
docker-commons:439.va_3cb_0a_6a_fb_29
docker-java-api:3.3.4-86.v39b_a_5ede342c
docker-plugin:1.5
durable-task:550.v0930093c4b_a_6
echarts-api:5.4.0-7
font-awesome-api:6.5.1-2
gcp-secrets-manager-credentials-provider:0.3.1
ghprb:1.42.2
git:5.2.1
git-client:4.6.0
github:1.37.3.1
github-api:1.318-461.v7a_c09c9fa_d63
github-branch-source:1767.va_7d01ea_c7256
github-checks:554.vb_ee03a_000f65
google-compute-engine:4.551.v5a_4dc98f6962
google-kubernetes-engine:0.424.v0a_79d1e437e0
google-metadata-plugin:0.5
google-oauth-plugin:1.321.ve7fa_e4c72544
google-source-plugin:0.4
google-storage-plugin:1.344.vc9a_c797ef183
gson-api:2.10.1-15.v0d99f670e0a_7
htmlpublisher:1.32
instance-identity:185.v303dc7c645f9
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.16.1-373.ve709c6871598
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jjwt-api:0.11.5-77.v646c772fddb_0
jquery3-api:3.7.1-1
jsch:0.2.16-86.v42e010d9484b_
json-path-api:2.9.0-33.v2527142f2e1d
junit:1259.v65ffcef24a_88
kubernetes:4054.v2da_8e2794884
kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
kubernetes-credentials:0.11
mailer:470.vc91f60c5d8e2
matrix-project:822.v01b_8c85d16d2
metrics:4.2.21-449.v6960d7c54c69
mina-sshd-api-common:2.12.0-90.v9f7fb_9fa_3d3b_
mina-sshd-api-core:2.12.0-90.v9f7fb_9fa_3d3b_
multibranch-scan-webhook-trigger:1.0.11
oauth-credentials:0.646.v02b_66dc03d2e
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
pipeline-aggregator-view:1.15
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-graph-view:205.vb_8e3a_b_51f12e
pipeline-groovy-lib:704.vc58b_8890a_384
pipeline-input-step:477.v339683a_8d55e
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2150.v4cfd8916915c
pipeline-model-definition:2.2150.v4cfd8916915c
pipeline-model-extensions:2.2150.v4cfd8916915c
pipeline-rest-api:2.34
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2150.v4cfd8916915c
pipeline-stage-view:2.34
plain-credentials:179.vc5cb_98f6db_38
plugin-util-api:3.8.0
prism-api:1.29.0-8
resource-disposer:0.23
saml:4.429.v9a_781a_61f1da_
scm-api:676.v886669a_199a_a_
script-security:1326.vdb_c154de8669
snakeyaml-api:2.2-111.vc6598e30cc65
ssh-credentials:322.v124df57ed808
ssh-slaves:2.948.vb_8050d697fec
sshd:3.303.vefc7119b_ec23
structs:337.v1b_04ea_4df7c8
test-results-analyzer:0.4.1
text-finder:1.26
timestamper:1.26
token-macro:400.v35420b_922dcb_
trilead-api:2.133.vfb_8a_7b_9c5dd1
variant:60.v7290fc0eb_b_cd
workflow-aggregator:596.v8c21c963d92d
workflow-api:1291.v51fd2a_625da_7
workflow-basic-steps:1047.v2e7111a_e4194
workflow-cps:3880.vb_ef4b_5cfd270
workflow-durable-task-step:1331.vc8c2fed35334
workflow-job:1385.vb_58b_86ea_fff1
workflow-multibranch:773.vc4fe1378f1d5
workflow-scm-step:415.v434365564324
workflow-step-api:657.v03b_e8115821b_
workflow-support:865.v43e78cc44e0d
ws-cleanup:0.45