Intermittent ExceptionInInitializerError and IllegalAccessError in durable-task under Java 21 with WebSocket, followed by an infinite agent reconnect loop
ENVIRONMENT AND ARCHITECTURE
Jenkins controller: 2.568.2, running in Kubernetes Cluster A
Agent pods: Kubernetes plugin ephemeral agents, running in Kubernetes Cluster B in a cross-cluster configuration
Agent image: jenkins/inbound-agent
Agent Java version: OpenJDK 21, Temurin-21.0.11+10
Transport: WebSocket enabled with JENKINS_WEB_SOCKET=true
Upgrade context: The issue appeared immediately after upgrading Jenkins from Java 11 to Java 21 and Jenkins 2.568.2.
Occurrence rate: Approximately 1 in 8,000 builds
Affected plugin versions:
durable-task:686.v80ff80875b_82
workflow-durable-task-step:1479.v56e587f413a_7
A potentially relevant observation is that Jenkins instances in the same environment using direct inbound TCP connections, without WebSocket enabled, have not reproduced this issue.
Due to the low occurrence rate, we have not yet been able to establish whether WebSocket or Java 21 is the direct cause. However, the issue has only been observed under this configuration.
When the issue occurs, the agent pod enters an unrecoverable reconnect loop until Kubernetes terminates it with DeadlineExceeded.
SYMPTOMS AND CONTROLLER BUILD LOG
A Pipeline sh step completes successfully. Exit code 0 is written to jenkins-result.txt on the agent filesystem in less than one second.
However, the Jenkins controller becomes trapped polling and reconnecting to the agent indefinitely:
11:30:18 [agent-pod] is back online
11:30:18 [agent-pod] seems to be removed or offline (java.io.IOException: Remote call on [agent-pod] failed); will wait for 5 min 0 sec for it to come back online
11:30:19 [agent-pod] is back online
11:30:19 [agent-pod] seems to be removed or offline (java.io.IOException: Remote call on [agent-pod] failed); will wait for 5 min 0 sec for it to come back online
This online/offline cycle continues until the pod reaches its configured deadline and is terminated by Kubernetes.
PRIMARY ERROR: EXCEPTIONININITIALIZERERROR CAUSED BY AN INTERRUPTED CLASS-LOADING OPERATION
In most affected runs, the agent’s jnlp log indicates that static initialization of FileMonitoringController failed while an in-flight remoting operation was interrupted:
WARNING: LinkageError while performing UserRequest:
org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController$TranscodingCharsetForSystemDefault@[object-id]
java.lang.NoClassDefFoundError: Could not initialize class org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController
at PluginClassLoader for durable-task//org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController$TranscodingCharsetForSystemDefault.call(FileMonitoringTask.java:380)
at PluginClassLoader for durable-task//org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController$TranscodingCharsetForSystemDefault.call(FileMonitoringTask.java:378)
at hudson.remoting.UserRequest.perform(UserRequest.java:225)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:391)
at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:81)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:312)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ExceptionInInitializerError: Exception hudson.remoting.RemotingSystemException: java.io.IOException: java.lang.InterruptedException [in thread “[remoting-thread-details-redacted]”]
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:328)
at jdk.proxy2/jdk.proxy2.$Proxy7.fetch3(Unknown Source)
at hudson.remoting.RemoteClassLoader.prefetchClassReference(RemoteClassLoader.java:363)
at hudson.remoting.RemoteClassLoader.loadWithMultiClassLoader(RemoteClassLoader.java:266)
at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:236)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
at PluginClassLoader for durable-task//org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.(FileMonitoringTask.java:411)
… 11 more
After this failure, subsequent attempts to use the same class result in:
java.lang.NoClassDefFoundError: Could not initialize class org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController
JVM CLASS-STATE INSPECTION
While the agent pod was stuck, we inspected the live JVM using:
jcmd 1 VM.classes
The output indicated that HotSpot had marked the affected class with an initialization error:
[class-address-redacted] 86 initialization_error fW org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController
This appears to make the agent JVM unrecoverable without restarting the pod.
SECONDARY ERROR: NESTMATE-RELATED ILLEGALACCESSERROR
In other occurrences of the same reconnect loop, when FileMonitoringController appeared to be fetched without an interruption, the agent reported the following nestmate-related IllegalAccessError under Java 21:
WARNING: LinkageError while performing UserRequest:
org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController$TranscodingCharsetForSystemDefault@[object-id]
java.lang.IllegalAccessError: class org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController$TranscodingCharsetForSystemDefault tried to access private method ‘java.nio.charset.Charset org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.transcodingCharset(java.lang.String)’
org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController$TranscodingCharsetForSystemDefault and org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController are in the unnamed module of loader ‘PluginClassLoader for durable-task’.
Nest host resolution of org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController$TranscodingCharsetForSystemDefault with host org/jenkinsci/plugins/durabletask/FileMonitoringTask$FileMonitoringController failed with:
java.lang.NoClassDefFoundError: org/jenkinsci/plugins/durabletask/FileMonitoringTask
at PluginClassLoader for durable-task//org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController$TranscodingCharsetForSystemDefault.call(FileMonitoringTask.java:380)
at PluginClassLoader for durable-task//org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController$TranscodingCharsetForSystemDefault.call(FileMonitoringTask.java:378)
COMMON BEHAVIOR AFTER EITHER ERROR
After either the ExceptionInInitializerError or the IllegalAccessError occurs:
1. The Pipeline shell command has already completed successfully.
2. The expected exit code is present in jenkins-result.txt.
3. The durable task monitoring operation fails.
4. The agent disconnects and reconnects through WebSocket.
5. Jenkins reports the agent as back online.
6. The next remote call fails again.
7. The agent enters an infinite reconnect loop.
8. Kubernetes eventually terminates the pod with DeadlineExceeded.
The agent does not recover after reconnecting because the class remains in an erroneous initialization state inside the same JVM.