SEVERE: Couldn't clean up oid=2 from null

Using Jenkins Ask a question
1

Jenkins setup:

Jenkins: 2.319.2
OS: Linux - 5.4.0-122-generic
Java: 11.0.14.1 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)

// Plugins skipped

Hi,

When running a swarm agent on our worker node, we observed the following errors:

Apr 10, 2024 8:54:51 PM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Apr 10, 2024 8:54:51 PM hudson.remoting.jnlp.Main createEngine
WARNING: Certificate validation for HTTPs endpoints is disabled
Apr 10, 2024 8:54:51 PM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 4.5
Apr 10, 2024 8:54:51 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using ./remoting as a remoting work directory
Apr 10, 2024 8:54:51 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to ./remoting
Apr 10, 2024 8:54:51 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [https://prod.waterfall.com/jenkins/]
Apr 10, 2024 8:54:51 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver openURLConnection
WARNING: HTTPs certificate check is disabled for the endpoint.
Apr 10, 2024 8:54:51 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Apr 10, 2024 8:54:51 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check
Apr 10, 2024 8:54:51 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
  Agent address: ptx-aws-waterfall-prod-k8ctl-06
  Agent port:    xxxxx
  Identity:      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Apr 10, 2024 8:54:51 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Apr 10, 2024 8:54:51 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to ptx-aws-waterfall-prod-k8ctl-06:32418
Apr 10, 2024 8:54:51 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Apr 10, 2024 8:54:51 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Apr 10, 2024 8:54:52 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Apr 10, 2024 8:54:54 PM hudson.remoting.RemoteInvocationHandler$Unexporter run
SEVERE: Couldn't clean up oid=2 from null
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:717)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
	at hudson.remoting.DelegatingExecutorService.submit(DelegatingExecutorService.java:48)
	at hudson.remoting.InterceptingExecutorService.submit(InterceptingExecutorService.java:46)
	at hudson.remoting.InterceptingExecutorService.submit(InterceptingExecutorService.java:41)
	at org.jenkinsci.remoting.util.AnonymousClassWarnings.check(AnonymousClassWarnings.java:66)
	at org.jenkinsci.remoting.util.AnonymousClassWarnings$1.annotateClass(AnonymousClassWarnings.java:122)
	at java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1290)
	at java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1231)
	at java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1294)
	at java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1231)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1427)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
	at hudson.remoting.Command.writeTo(Command.java:111)
	at hudson.remoting.AbstractByteBufferCommandTransport.write(AbstractByteBufferCommandTransport.java:287)
	at hudson.remoting.Channel.send(Channel.java:766)
	at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.cleanup(RemoteInvocationHandler.java:398)
	at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.access$1000(RemoteInvocationHandler.java:357)
	at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:615)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:118)
	at java.lang.Thread.run(Thread.java:748)

It looks like the worker node ran out of native JVM threads, but what does this mean?

SEVERE: Couldn't clean up oid=2 from null

Thanks for any insights!

2

There are comments in JENKINS-65873 that changes were made in Jenkins remoting for Jenkins 2.361.1 and later that can address the failure to create native threads. Jenkins 2.319.2 was released over two years ago. It has known security vulnerabilities. You should upgrade to the most recent Jenkins release so that you get the latest fixes and improvements.

There are comments in JENKINS-68199 that newer releases of Java 11 are believed to reduce the incidence of the failure to create a native thread. Java 11.0.23 is the current patch level. Java is patched every 3 months with new fixes. Java 11.0.14 is two years old. Upgrade to Java 11.0.23. Even if you can’t upgrade to a newer Jenkins version, I think that you should upgrade your Java version.

1 Like
3

@MarkEWaite The 2 threads are very helpful. Thank you very much for such detailed response and for your time. I will look into upgrading our Jenkins controller and the hudson remoting lib.