Memory Leak Issues

Using Jenkins
1

Hey Experts,
We are using Jenkins 2.246.3, Java 11.0.23.
After the last plugin upgrade, we are seeing the JVM Heap memory being accumulated steadily.

used-memory
used-memory1688Ă—842 149 KB

swap-space
swap-space1688Ă—842 124 KB

How can we investigate if any one of the plugins is causing a memory leak

Plugin(Version :From)                               Version :To  
bouncycastle-api (2.29)                             2.30.1.77-225.v26ea_c9455fd9
config-file-provider (3.8.2)                        953.v0432a_802e4d2
jackson2-api (2.15.3-366.vfe8d1fa_f8c87)            2.16.1-373.ve709c6871598
ldap (1.13)                                         725.v3cb_b_711b_1a_ef
pipeline-graph-analysis (1.2)                       202.va_d268e64deb_3
pipeline-input-step (2.12)                          468.va_5db_051498a_4   
pipeline-model-api (1.9.2)                          2.2118.v31fd5b_9944b_5 
pipeline-model-definition (1.9.2)                   2.2118.v31fd5b_9944b_5 
pipeline-model-extensions (1.9.2)                   2.2118.v31fd5b_9944b_5 
pipeline-stage-tags-metadata (1.9.2)                2.2118.v31fd5b_9944b_5 
pipeline-rest-api (2.9)                             2.34   
pipeline-stage-step (2.5)                           305.ve96d0205c1c6  
role-strategy (3.1.1)                               717.v6a_69a_fe98974
workflow-basic-steps (2.23)                         969.vc4ec3e4854b_f 
okhttp-api (3.14.9)                                 4.9.3-108.v0feda04578cf
saml (1.1.7)                                        4.464.vea_cb_75d7f5e0  
metrics (4.0.2.8.1)                                 4.2.18-442.v02e107157925   
jaxb (2.3.8-1)                                      2.3.9-1
javax-mail-api (1.6.2-2)                            1.6.2-6
jakarta-activation-api (2.0.1-1)                    2.0.1-3
jakarta-mail-api (2.0.1-1)                          2.0.1-3
json-path-api (New-Plugin)                          2.8.0-5.v07cb_a_1ca_738c   
prometheus (New-Plugin)                             787.v52e8f47488fc
joda-time-api (New-Plugin)                          2.12.5-5.v5495a_235fedf
2

First you need to get the heap dump(.hprof): There are several ways to go about this.

  • Run JVM with -XX:+HeapDumpOnOutOfMemoryError so that JVM will automatically produce a heap dump when it hits OutOfMemoryError.
  • You can run jmap -dump:live,file=/tmp/jenkins.hprof pid where pid is the process ID of the target Java process.
  • Use VisualVM, attach to the running instance, and obtain a heap dump
  • If your Jenkins runs at http: //server/jenkins/, request http: //server/jenkins/heapDump with your browser and you’ll get the heap dump downloaded. (1.395 and newer)
  • If you are familiar with one of many Java profilers, they normally offer this capability, too.

Then you need to do a Heap Dump Analysis with a tool like Eclipse Memory Analyzer you can then check the suspects and view the java classes to find the culprit.

I recommend using a powerful machine to run MAT if your heap dump is large. By default, MAT is limited to 1GB of memory usage, so I suggest adjusting it to match your machine’s available capacity by modifying the “-Xmx” flag in the “MemoryAnalyzer.ini” configuration file. Once you load the heap dump, be prepared to wait, as it can take a significant amount of time to load and analyze.

1 Like