Hey Experts,
We have Jenkins Core: 2.426.3 Tomcat: 9.0.56 Kubernetes Plugin: 1.30.11 Java: 11.0.16.1
Server Used: OL8
Recently we have been observing issues with Jenkins JNLP Connection (intermittently, but more often) issues;
The pod created Terminates on
INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent
at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
at hudson.remoting.Engine.innerRun(Engine.java:778)
at hudson.remoting.Engine.run(Engine.java:540)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent
at org.jenkinsci.remoting.protocol.impl.AckFilterLayer.onRecvClosed(AckFilterLayer.java:280)
at org.jenkinsci.remoting.protocol.FilterLayer.abort(FilterLayer.java:164)
at org.jenkinsci.remoting.protocol.impl.AckFilterLayer.lambda$start$0(AckFilterLayer.java:177)
at org.jenkinsci.remoting.protocol.IOHub$DelayedRunnable.run(IOHub.java:959)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:122)
at java.base/java.lang.Thread.run(Thread.java:829)
Apr 12, 2024 4:43:40 PM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: The server rejected the connection: None of the protocols were accepted
java.lang.Exception: The server rejected the connection: None of the protocols were accepted
at hudson.remoting.Engine.onConnectionRejected(Engine.java:864)
at hudson.remoting.Engine.innerRun(Engine.java:804)
at hudson.remoting.Engine.run(Engine.java:540)
The JNLP image used: jnlp-agent:latest-jdk11 ( Remoting Version: 4.9, we are using a suppression to allow this remoting agent hudson.slaves.SlaveComputer.allowUnsupportedRemotingVersions=true )
Usually, rebooting the VM fixes the issue. But, we are looking for a much stable solution.
During the issue, Jenkins Container Logs have
12-Apr-2024 16:41:47.957 SEVERE [OkHttp Dispatcher] hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler.uncaughtException A thread (OkHttp Dispatcher/28257481) died unexpectedly due to an uncaught exception. This may leave your server corrupted and usually indicates a software bug.
12-Apr-2024 16:41:48.004 SEVERE [OkHttp Dispatcher] hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler.uncaughtException A thread (OkHttp Dispatcher/28257486) died unexpectedly due to an uncaught exception. This may leave your server corrupted and usually indicates a software bug.
12-Apr-2024 16:41:49.008 SEVERE [TCP agent listener port=50000] hudson.TcpSlaveAgentListener.run Failed to accept TCP connections
12-Apr-2024 16:41:55.209 SEVERE [TCP agent listener port=50000] hudson.TcpSlaveAgentListener.run Failed to accept TCP connections
12-Apr-2024 16:41:56.721 SEVERE [TCP agent listener port=50000] hudson.TcpSlaveAgentListener.run Failed to accept TCP connections
We have observed that the memory and CPU usage are within the threshold, and the load on Jenkins is optimum.
Apr 12, 2024 4:29:26 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [https://<jenkins-dns>:<port>/jenkins/]
Apr 12, 2024 4:29:26 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Apr 12, 2024 4:29:26 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
Agent address: <jenkins-dns>
Agent port: 50000
Identity: 27:43:04:cb:d5:23:81:ef:d9:4a:94:bf:67:ad:b4:29
Apr 12, 2024 4:29:26 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Apr 12, 2024 4:29:26 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to <jenkins-dns>:50000
Apr 12, 2024 4:29:26 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Apr 12, 2024 4:29:26 PM org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader run
INFO: Waiting for ProtocolStack to start.
Apr 12, 2024 4:29:37 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent
at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
at hudson.remoting.Engine.innerRun(Engine.java:822)
at hudson.remoting.Engine.run(Engine.java:543)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent
at org.jenkinsci.remoting.protocol.impl.AckFilterLayer.onRecvClosed(AckFilterLayer.java:280)
at org.jenkinsci.remoting.protocol.FilterLayer.abort(FilterLayer.java:165)
at org.jenkinsci.remoting.protocol.impl.AckFilterLayer.lambda$start$0(AckFilterLayer.java:177)
at org.jenkinsci.remoting.protocol.IOHub$DelayedRunnable.run(IOHub.java:958)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125)
at java.base/java.lang.Thread.run(Thread.java:834)
Apr 12, 2024 4:29:37 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: reconnect rejected, sleeping 10s:
java.lang.Exception: The server rejected the connection: None of the protocols were accepted
at hudson.remoting.Engine.onConnectionRejected(Engine.java:901)
at hudson.remoting.Engine.innerRun(Engine.java:848)
at hudson.remoting.Engine.run(Engine.java:543)
I don’t see any container image in the Jenkins organization called jnlp-agent. I see a container image named jnlp-slave that is deprecated and has not been updated for over a year. That page says:
Using a current agent container image is better than using a 3 year old agent container image.
Using an unsupported remoting version won’t help with the reliability of the connection between the agent and the controller. You’re much more likely to have a good experience and to receive help from others if you use the most recent agent container images.
Update to use a more recent version of the Jenkins agent container so that you’re not running an agent with a 3 year old Java version and a 3 year old base operating system and an unsupported configuration.
Theoretically, Jenkins can also be run as a servlet in a traditional servlet container like Apache Tomcat or WildFly. However, in practice, this is largely untested, and there are many caveats. …
Support for traditional servlet containers may be discontinued in the future.
We did use inbound-agent:latest-alpine3.19-jdk11 initially. When we ran into a similar issue with the inbound-agent, we moved back to use jnlp-slave with a lower remoting version.
@MarkEWaite Will try to upgrade TomCat to 9.0.85 and Java to 11.0.23!
Including this section in server.xml
Scheme selection in redirect URLs is delegated to the servlet container, and Jetty handles the X-Forwarded-For , X-Forwarded-By , and X-Forwarded-Proto headers by default. With Tomcat, one needs to add a Remote IP Valve to expose these headers to Jenkins via the Servlet API. Add the following to ${CATALINA_HOME}/conf/server.xml within the <Host> section:
You’ve chosen to run in a configuration that Jenkins core maintainers do not test. I think that choice means that you’ve also chosen to be the one who can validate if that improves the issue.
I recommend that you switch from using Tomcat to instead use the web container that is provided in the Jenkins WAR file. If you want a more stable solution, then you should choose to use a configuration that is more widely used, better tested, and better supported. The web container that is included in the Jenkins WAR
Is there a way to validate if this improves the issue seen?