How to identify a thread leak issue and determine which builds are contributing to it?

Using Jenkins Ask a question
1

The Jenkins service is running version 2.426.1 and is in production usage, utilized by multiple teams.
The service intermittently throws an error: ‘java.lang.OutOfMemoryError: unable to create native thread,’ causing it to stop working as expected. Upon restart, it functions normally for a few days (typically 4 to 5 days) before encountering the same error again.
Initial investigation reveals that the service crashes when it reaches its maximum allowable tasks (Java threads), which is 4915.
After collecting a thread dump and analyzing it with tools, the following stack trace output was obtained.

Before restart:

Count Line
4849 java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
4595 java.base@11.0.16/java.lang.Thread.sleep(Native Method)
4583 org.tmatesoft.svn.core.internal.io.svn.StreamLogger$$Lambda$896/0x00007f000b4e5960.run(Unknown Source)
4583 org.tmatesoft.svn.core.internal.io.svn.StreamLogger.lambda$new$0(StreamLogger.java:58)
229 java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
160 java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)

After restart:

Count Line
143 java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
99 java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
76 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
76 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
52 java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
47 java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)

It has been observed that the sleep thread (major number) is being terminated after the service restarts.

Assistance is needed to identify this thread leak issue and determine which build or job is causing it. Thank you in advance.

2

https://svn.tmatesoft.com/repos/svnkit/tags/1.10.10/svnkit/src/main/java/org/tmatesoft/svn/core/internal/io/svn/StreamLogger.java starts a thread. StreamLogger also implements Closeable, and its close() method would interrupt and join the thread. It seems close() is not called, then. I don’t know whose responsibility that would be; what even causes the StreamLogger instance to be created.

The Subversion plugin for Jenkins uses svnkit. However, its source code does not directly mention StreamLogger. It is possible that the StreamLogger instance is created by some other class in svnkit, and that the Subversion plugin would have to call the close() method of that class in order to close the StreamLogger too.

If you can use a Java debugger to set a breakpoint at the org.tmatesoft.svn.core.internal.io.svn.StreamLogger constructor and get a stack trace from there, I think it will help figure this out.

3

Thank you for your reply, and I appreciate your insights on this issue.

Could you please provide more details on using the Java debugger, considering that the current system is in production.
Is it possible to identify which job or build is triggering the issue.
Surprisingly, the Subversion plugin is in a disabled state.

Your assistance on this would be greatly appreciated. Thank you once again for your attention to this issue.

4

I’m not a Java programmer by trade and cannot advise on how to use Java debuggers.

If the problem occurs in production, then perhaps there is some logger that one can enable to get an idea of what triggers the thread creation. But I’m not sure how to find one.

Are you using Subversion with your Jenkins controller in some manner or is the use of svnkit entirely unexpected?

5

Thank you for the update on the Java debugger.
The Subversion plugin is installed, but it’s currently in a disabled state.
I couldn’t find svnkit in the list of installed plugins on the system. As Subversion is our repository server and is heavily utilized in pipeline scripts and SCM polling schedules.