Jenkins setup:
Jenkins: 2.509
OS: Linux - 4.14.336-257.562.amzn2.x86_64
Java: 21.0.7 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
Parameterized-Remote-Trigger:3.2.1
amazon-ecr:1.151.vb_ca_71ddd0b_cf
amazon-ecs:1.49
analysis-model-api:13.2.0
ansicolor:1.0.6
ant:513.vde9e7b_a_0da_0f
antisamy-markup-formatter:173.v680e3a_b_69ff3
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
apache-httpcomponents-client-5-api:5.4.4-144.vc483d91903d2
artifact-manager-s3:902.v39b_3d56b_88b_5
artifactory:4.0.8
asm-api:9.8-135.vb_2239d08ee90
audit-trail:395.vce180b_359a_b_5
authentication-tokens:1.131.v7199556c3004
authorize-project:2.0.0
aws-codebuild:0.59
aws-credentials:245.v8a_1b_7c11a_94d
aws-global-configuration:146.vfcec61593eea_
aws-java-sdk:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-api-gateway:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-autoscaling:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudformation:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudfront:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudwatch:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-codebuild:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-codedeploy:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ec2:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ecr:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ecs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-efs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-elasticbeanstalk:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-elasticloadbalancingv2:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-iam:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-kinesis:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-lambda:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-logs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-minimal:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-organizations:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-secretsmanager:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-sns:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-sqs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ssm:1.12.780-480.v4a_0819121a_9e
aws-java-sdk2-core:2.31.26-37.vd178d9b_0558f
aws-java-sdk2-ec2:2.31.26-37.vd178d9b_0558f
aws-java-sdk2-ecr:2.31.26-37.vd178d9b_0558f
aws-parameter-store:1.2.2
basic-branch-build-strategies:228.v68c089762a_db_
blueocean:1.27.19
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.19
blueocean-commons:1.27.19
blueocean-config:1.27.19
blueocean-core-js:1.27.19
blueocean-dashboard:1.27.19
blueocean-display-url:2.4.4
blueocean-events:1.27.19
blueocean-git-pipeline:1.27.19
blueocean-github-pipeline:1.27.19
blueocean-i18n:1.27.19
blueocean-jira:1.27.19
blueocean-jwt:1.27.19
blueocean-personalization:1.27.19
blueocean-pipeline-api-impl:1.27.19
blueocean-pipeline-editor:1.27.19
blueocean-pipeline-scm-api:1.27.19
blueocean-rest:1.27.19
blueocean-rest-impl:1.27.19
blueocean-web:1.27.19
bootstrap5-api:5.3.5-1
bouncycastle-api:2.30.1.80-256.vf98926042a_9b_
branch-api:2.1217.v43d8b_b_d8b_2c7
build-discarder:158.vce570d01ce4c
build-failure-analyzer:2.5.5
build-timeout:1.38
build-token-root:151.va_e52fe3215fc
build-user-vars-plugin:195.v8c35f9d5c3dc
build-with-parameters:76.v9382db_f78962
caffeine-api:3.2.0-166.v72a_6d74b_870f
checks-api:370.vb_61a_c57328f3
cloudbees-bitbucket-branch-source:936.1.1
cloudbees-folder:6.1012.v79a_86a_1ea_c1f
code-coverage-api:4.7.0
command-launcher:123.v37cfdc92ef67
commons-compress-api:1.27.1-3
commons-lang3-api:3.17.0-87.v5cf526e63b_8b_
commons-text-api:1.13.0-153.v91dcd89e2a_22
compact-columns:1.199.v61a_f51712072
config-file-provider:988.v0461fcc2b_9d1
configuration-as-code:1967.va_968e15fd05b_
configuration-as-code-secret-ssm:1.0.2-SUMO
copyartifact:770.va_6c69e063442
credentials:1415.v831096eb_5534
credentials-binding:687.v619cb_15e923f
data-tables-api:2.2.2-1
disable-github-multibranch-status:1.2
display-url-api:2.209.v582ed814ff2f
docker-commons:451.vd12c371eeeb_3
docker-workflow:611.v16e84da_6d3ff
durable-task:587.v84b_877235b_45
ec2:1911.vccb_5a_56f62e6
ec2-fleet:3.2.0
echarts-api:5.6.0-4
eddsa-api:0.3.0.1-19.vc432d923e5ee
email-ext:1876.v28d8d38315b_d
extended-read-permission:3.2
external-monitor-job:223.vb_fddcf42c9b_3
favorite:2.225.v68765b_b_a_1fa_3
font-awesome-api:6.7.2-1
forensics-api:3.1.0
generic-webhook-trigger:2.3.1
git:5.7.0
git-client:6.1.3
git-forensics:3.2.0
git-server:137.ve0060b_432302
github:1.43.0
github-api:1.321-488.v9b_c0da_9533f8
github-branch-source:1815.v9152b_2ff7a_1b_
github-checks:602.v264a_83610da_6
github-pr-comment-build:134.ve7ff0b_719821
gradle:2.14.1
groovy:497.v7b_061a_a_de65d
gson-api:2.13.1-139.v4569c2ef303f
handy-uri-templates-2-api:2.1.8-36.v85e4cb_234a_13
hidden-parameter:414.vfe0a_8b_052546
htmlpublisher:425
http_request:1.20
inline-pipeline:1.0.3
instance-identity:203.v15e81a_1b_7a_38
ionicons-api:88.va_4187cb_eddf1
jackson2-api:2.18.3-402.v74c4eb_f122b_2
jakarta-activation-api:2.1.3-2
jakarta-mail-api:2.1.3-2
javadoc:327.vdfe586651ee0
javax-activation-api:1.2.0-8
javax-mail-api:1.6.2-11
jaxb:2.3.9-133.vb_ec76a_73f706
jdk-tool:83.v417146707a_3d
jenkins-design-language:1.27.19
jersey2-api:2.45-154.v4ded3dc34f81
jira:3.15
jjwt-api:0.11.5-120.v0268cf544b_89
jmh-report:0.9.0
jnr-posix-api:3.1.20-138.vdb_9db_a_39182f
job-dsl:1.92
jobConfigHistory:1305.vf20a_356586b_8
joda-time-api:2.14.0-127.v7d9da_295a_d51
jquery:1.12.4-1
jquery3-api:3.7.1-3
jsch:0.2.16-95.v3eecb_55fa_b_78
json-api:20250107-125.v28b_a_ffa_eb_f01
json-path-api:2.9.0-148.v22a_7ffe323ce
jsoup:1.20.1-46.ve5f1416988c2
junit:1322.v1556dc1c59a_f
ldap:780.vcb_33c9a_e4332
lockable-resources:1166.v7ef984f72fe2
mailer:489.vd4b_25144138f
mapdb-api:1.0.9-44.va_1e1310c9118
matrix-auth:3.2.6
matrix-project:849.v0cd64ed7e531
maven-plugin:3.26
mercurial:1260.vdfb_723cdcc81
metrics:4.2.30-471.v55fa_495f2b_f5
mina-sshd-api-common:2.15.0-161.vb_200831a_c15b_
mina-sshd-api-core:2.15.0-161.vb_200831a_c15b_
mina-sshd-api-scp:2.15.0-161.vb_200831a_c15b_
monitoring:2.5.0
multibranch-build-strategy-extension:61.vf6d8f6f5ed02
nested-view:1.33
next-executions:424.v631378313e29
node-iterator-api:72.vc90e81737df1
okhttp-api:4.11.0-189.v976fa_d3379d6
oss-symbols-api:324.v432cce4172ca_
pam-auth:1.12
parameterized-scheduler:285.ve611986d4c48
pipeline-aws:1.45
pipeline-build-step:567.vea_ce550ece97
pipeline-github:2.8-162.382498405fdc
pipeline-graph-analysis:235.vb_a_a_36b_f248c2
pipeline-graph-view:514.vc1792344c64f
pipeline-groovy-lib:752.vdddedf804e72
pipeline-input-step:517.vf8e782ee645c
pipeline-milestone-step:134.vdf60d179845f
pipeline-model-api:2.2255.v56a_15e805f12
pipeline-model-definition:2.2255.v56a_15e805f12
pipeline-model-extensions:2.2255.v56a_15e805f12
pipeline-rest-api:2.38
pipeline-stage-step:322.vecffa_99f371c
pipeline-stage-tags-metadata:2.2255.v56a_15e805f12
pipeline-stage-view:2.38
pipeline-utility-steps:2.19.0
plain-credentials:195.vb_906e9073dee
plugin-util-api:6.1.0
prism-api:1.30.0-1
pubsub-light:1.19
resource-disposer:0.25
reverse-proxy-auth-plugin:238.v82ceca_8417a_6
role-strategy:756.v978cb_392eb_d3
run-condition:1.3
saml:4.525.v4f6a_7209447e
scm-api:704.v3ce5c542825a_
scoverage:1.4.0
script-security:1373.vb_b_4a_a_c26fa_00
scriptler:363.vd97ef616cb_f9
show-build-parameters:1.0
simple-theme-plugin:211.v5424a_5510e47
slack:761.v2a_8770f0d169
snakeyaml-api:2.3-125.v4d77857a_b_402
snyk-security-scanner:4.1.0
sse-gateway:1.28
ssh-agent:384.ve275343791a_6
ssh-credentials:355.v9b_e5b_cde5003
ssh-slaves:3.1031.v72c6b_883b_869
sshd:3.353.v2b_d33c46e970
structs:343.vdcf37b_a_c81d5
subversion:1287.vd2d507146906
sumologic-publisher:2.2.6
support-core:1725.va_2a_9f06eed61
throttle-concurrents:2.16
timestamper:1.28
token-macro:444.v52de7e9c573d
trilead-api:2.209.v0e69b_c43c245
uno-choice:2.8.3
variant:70.va_d9f17f859e0
view-job-filters:399.v6c89d5e6ff54
warnings-ng:12.5.0
windows-slaves:1.8.1
workflow-aggregator:608.v67378e9d3db_1
workflow-api:1371.ve334280b_d611
workflow-basic-steps:1079.vce64b_a_929c5a_
workflow-cps:4106.v7a_8a_8176d450
workflow-durable-task-step:1405.v1fcd4a_d00096
workflow-job:1520.v56d65e3b_4566
workflow-multibranch:806.vb_b_688f609ee9
workflow-scm-step:437.v05a_f66b_e5ef8
workflow-step-api:700.v6e45cb_a_5a_a_21
workflow-support:968.v8f17397e87b_8
ws-cleanup:0.48
We’ve been observing frequent instances where Jenkins becomes unresponsive. During these occurrences, the Jenkins instance appears heavily hung, with a significant number of active threads.
The heap was initially configured with 16GB. As memory usage continued to grow, we increased it to 18GB and later to 20GB, but the issue still persisted.
We attempted to analyze the thread dumps to identify and terminate any problematic threads, but that didn’t resolve the issue. Ultimately, the only reliable way to recover Jenkins has been to restart the instance.
However, a safeRestart does not help in this scenario. We consistently need to go to the AWS console, update the service, and trigger a new deployment to restore Jenkins functionality.
Took a heap dump and analyzed it through Eclipse MAT.
1. Overall Heap Breakdown * Total heap inspected: ~9.4 GB
2. Analyzed the below two suspects WorkflowRun objects (2.97GB) and CpsFlowExecution(2.30GB):
(a) WorkflowRun objects (~2.97 GB):
(b) CpsFlowExecution objects (~2.30 GB)
Started analyzing the source of these WorkflowRun objects and found that top contributors were different jobs from the affected jenkins.
How do we ensure these leaks be removed? Any suggestions/help is highly appreciated.