I am running Jenkins 2.401 on Kubernetes. The controller runs on JDK 11. The setup makes use of 2 inbound agents to run different pipeline jobs.
inbound agent 1 uses 3198.v03a_401881f3e-1-jdk17 docker image (JDK - 17)
inbound agent 2 uses 3107.v665000b_51092-15. docker image (JDK - 11)
We are intermittently seeing nodes turning offline when the job begins when using both the agents. There is no pattern observed, the error may appear after 15 pipeline submissions or may not appear for 30 submissions.
Below is the message written to the controller log file. The agent log file remains clear with no errors.
2024-10-01 04:51:46.333+0000 [id=7688] WARNING j.m.api.Metrics$HealthChecker#execute: Some health checks are reporting as unhealthy: [thread-deadlock : [Computer.threadPoolForRemoting [#1] locked on java.util.concurrent.locks.ReentrantLock$NonfairSync@4acb9e4c (owned by jenkins.util.Timer [#1]): at java.base@11.0.19/jdk.internal.misc.Unsafe.park(Native Method)
I came across a recommendation in Jenkins JIRA to upgrade the pipeline plugin, however the version of the plugin we are running is latest.
We have plans to upgrade Jenkins and agents to run on JDK - 17 starting next year, However we are unable to determine the cause of the problem.
When this happens, restarting Jenkins from the console does not work. Aborting a job does not terminate the node/pod, however the job gets aborted. Getting into nodes and cloud to delete the node, does not get the node deleted.
The agent pod does not report any errors.
Temporarily we have disabled the node version monitor plugin, the workaround currently in place is to scale down Jenkins controller to 0 and scale it back to 1.
The Kubernetes version 1.30.
Reproduction steps
No steps to reproduce as the issue does not have any pattern.
Jenkins setup:
Jenkins: 2.401.3
OS: Linux - 6.1.100+
Java: 11.0.19 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
JDK_Parameter_Plugin:1.2
ace-editor:1.1
ansible:253.v4fe719ffdd8a_
ansicolor:1.0.4
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
apache-httpcomponents-client-5-api:5.4-124.v31e2987e48f4
audit-log:1.3
audit-trail:361.v82cde86c784e
authentication-tokens:1.119.v50285141b_7e1
authorize-project:1.7.1
bitbucket-build-status-notifier:1.4.2
blueocean-bitbucket-pipeline:1.27.11
blueocean-commons:1.27.11
blueocean-core-js:1.27.11
blueocean-jwt:1.27.11
blueocean-pipeline-api-impl:1.27.11
blueocean-pipeline-scm-api:1.27.11
blueocean-rest:1.27.11
blueocean-rest-impl:1.27.11
blueocean-web:1.27.11
bootstrap4-api:4.6.0-6
bootstrap5-api:5.3.2-3
bouncycastle-api:2.30.1.78.1-248.ve27176eb_46cb_
branch-api:2.1128.v717130d4f816
build-user-vars-plugin:166.v52976843b_435
build-with-parameters:76.v9382db_f78962
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.0.2
cloud-stats:336.v788e4055508b_
cloudbees-bitbucket-branch-source:888.v8e6d479a_1730
cloudbees-folder:6.858.v898218f3609d
command-launcher:107.v773860566e2e
commons-httpclient3-api:3.1-3
commons-lang3-api:3.17.0-84.vb_b_938040b_078
commons-text-api:1.11.0-108.v2c41b_8b_2b_269
conditional-buildstep:1.4.3
configuration-as-code:1700.v6f448841296e
copyartifact:722.v0662a_9b_e22a_c
credentials:1319.v7eb_51b_3a_c97b_
credentials-binding:642.v737c34dea_6c2
dark-theme:336.v02165cd8c2ee
data-tables-api:1.13.6-5
datadog:7.2.1
date-parameter:0.0.4
display-url-api:2.204.vf6fddd8a_8b_e9
docker-commons:443.v921729d5611d
docker-java-api:3.4.0-94.v65ced49b_a_7d5
docker-plugin:1.5
docker-workflow:580.vc0c340686b_54
durable-task:555.v6802fe0f0b_82
echarts-api:5.4.0-7
extended-read-permission:53.v6499940139e5
favorite:2.4.3
font-awesome-api:6.5.1-2
gcp-secrets-manager-credentials-provider:0.3.1
generic-webhook-trigger:2.0.1
git:5.2.1
git-client:4.6.0
git-server:126.v0d945d8d2b_39
github:1.37.3.1
github-api:1.318-461.v7a_c09c9fa_d63
github-branch-source:1771.v59b_6a_fa_1b_89e
google-metadata-plugin:0.5
google-oauth-plugin:1.330.vf5e86021cb_ec
google-storage-plugin:1.360.v6ca_38618b_41f
gson-api:2.10.1-15.v0d99f670e0a_7
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
hashicorp-vault-pipeline:1.4
hashicorp-vault-plugin:371.v884a_4dd60fb_6
hidden-parameter:138.v2a_c30d04e566
htmlpublisher:1.36
instance-identity:201.vd2a_b_5a_468a_a_6
ionicons-api:74.v93d5eb_813d5f
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javax-activation-api:1.2.0-7
javax-mail-api:1.6.2-10
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jenkins-design-language:1.27.11
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
jnr-posix-api:3.1.19-1
jobConfigHistory:1277.vb_9a_0808495d7
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery-detached:1.2.1
jquery3-api:3.7.1-1
jsch:0.2.16-86.v42e010d9484b_
json-api:20240205-27.va_007549e895c
json-path-api:2.9.0-33.v2527142f2e1d
junit:1265.v65b_14fa_f12f0
kubernetes:4054.v2da_8e2794884
kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
kubernetes-credentials:174.va_36e093562d9
lockable-resources:1245.vb_05f8a_4e28db_
login-theme:46.v36f624efb_23d
mailer:470.vc91f60c5d8e2
material-theme:0.5.2-rc100.6121925fe229
matrix-auth:3.2.2
matrix-project:822.824.v14451b_c0fd42
mercurial:1260.vdfb_723cdcc81
metrics:4.2.21-451.vd51df8df52ec
metrics-datadog:1.0
mina-sshd-api-common:2.12.1-101.v85b_e08b_780dd
mina-sshd-api-core:2.12.1-101.v85b_e08b_780dd
miniorange-saml-sp:2.4.4
momentjs:1.1.1
multiple-scms:0.8
multiselect-parameter:1.7
oauth-credentials:0.653.v14cf2088e950
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
packer:1.5
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-groovy-lib:689.veec561a_dee13
pipeline-input-step:477.v339683a_8d55e
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2150.v4cfd8916915c
pipeline-model-definition:2.2150.v4cfd8916915c
pipeline-model-extensions:2.2150.v4cfd8916915c
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2150.v4cfd8916915c
pipeline-stage-view:2.34
pipeline-utility-steps:2.16.0
plain-credentials:182.v468b_97b_9dcb_8
plugin-util-api:3.8.0
popper-api:1.16.1-3
popper2-api:2.11.6-4
pubsub-light:1.18
purge-build-queue-plugin:88.v23b_97b_f2c7a_d
rebuild:332.va_1ee476d8f6d
resource-disposer:0.24
role-strategy:689.v731678c3e0eb_
run-condition:1.7
saferestart:0.7
saml:4.464.vea_cb_75d7f5e0
sbt:81.vb_82499046630
scm-api:676.v886669a_199a_a_
script-security:1365.v4778ca_84b_de5
simple-theme-plugin:160.vb_76454b_67900
slack:684.v833089650554
snakeyaml-api:2.3-123.v13484c65210a_
solarized-theme:0.1
ssh-agent:367.vf9076cd4ee21
ssh-credentials:337.v395d2403ccd4
ssh-slaves:2.948.vb_8050d697fec
ssh-steps:2.0.68.va_d21a_12a_6476
sshd:3.322.v159e91f6a_550
structs:325.vcb_307d2a_2782
terraform:1.0.10
theme-manager:211.vef2a_42c645a_b_
thinBackup:2.0
timestamper:1.27
token-macro:400.v35420b_922dcb_
trilead-api:2.142.v748523a_76693
uno-choice:2.8.1
variant:60.v7290fc0eb_b_cd
versioncolumn:233.v2d198f8212a_2
windows-slaves:1.8.1
workflow-aggregator:600.vb_57cdd26fdd7
workflow-api:1291.v51fd2a_625da_7
workflow-basic-steps:1058.vcb_fc1e3a_21a_9
workflow-cps:3837.v305192405b_c0
workflow-cps-global-lib:609.vd95673f149b_b
workflow-durable-task-step:1331.vc8c2fed35334
workflow-job:1326.ve643e00e9220
workflow-multibranch:770.v1a_d0708dd1f6
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:907.v6713a_ed8a_573
ws-cleanup:0.46