Solved: Jenkins LTS Spawning Threads Until Unusable

Hello,
We are seeing an issue where we have a Jenkins instance with ~2200 jobs that had been generally stable. We are running on CentOS 7.9 and updated to 2.361.1 with Java 11 a few weeks ago and noticed that after a few days Jenkins would start to slow down, eventually becoming nearly unresponsive. We updated to 2.361.4, and still see the same issue.
Using the “Monitoring” plugin, we see the thread count after a restart sit around 1,000, and then slowly climb. Once it gets to ~18,000-20,000 threads, the system is unusable.

Looking at the threads, they all seem to be a non-descript “Thread-xxxxx”.


With the tooltip saying something like:
Thread-1152
java.base@11.0.7/java.lang.Object.wait(Native Method)
java.base@11.0.7/java.lang.Object.wait([Object.java:328])
java.naming@11.0.7/com.sun.jndi.ldap.Connection.pauseReader([Connection.java:759])
java.naming@11.0.7/com.sun.jndi.ldap.Connection.run([Connection.java:899])
java.base@11.0.7/java.lang.Thread.run([Thread.java:834])

Similar issues in the past from the forums have pointed to rogue plugins, but this does not seem to point directly to any one plugin.

Any help is appreciated.

Eventually found a bug report from searching for the error messages in the thread tooltip.
https://issues.jenkins.io/browse/JENKINS-52374?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel

We were using Java 11.0.7, and the above issue mentioned a fix in Java 11.0.8. We upgraded to that, restarted Jenkins, and now no long see the Thread-xxxx in a waiting state.

Thread count has remained stable for 24 hours.