Windows Jenkins agents are disconnecting and tunneling port stopped accepting requests

Using Jenkins
, ,
1

Hi, I’m using Jenkins which
Jenkins version is JENKINS_VERSION 2.375.2
Java version is java.class.version 55.0
https.protocols TLSv1.2
The master node is deployed on an ubuntu server (4.15.0-142-generic #146~16.04.1-Ubuntu)
And connected to agents through tunneling on port 50000, agents are windows and connected to the master using the jnlp command.
Now, all of a sudden some agents disconnect for the master and when trying to reconnect gives me this error.

May 14, 2023 12:33:47 PM hudson.remoting.jnlp.Main$CuiListener status                                                   INFO: Connecting to <jenkins>:50000 (retrying:7)
java.io.IOException: Failed to connect to <jenkins>:50000 
at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:248)                                     at hudson.remoting.Engine.connectTcp(Engine.java:900) 
at hudson.remoting.Engine.innerRun(Engine.java:784) 
at hudson.remoting.Engine.run(Engine.java:543)
Caused by: java.net.ConnectException: Connection refused: connect
at java.base/sun.nio.ch.Net.connect0(Native Method) 
at java.base/sun.nio.ch.Net.connect(Net.java:483) 
at java.base/sun.nio.ch.Net.connect(Net.java:472)
at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:692) 
at java.base/java.nio.channels.SocketChannel.open(SocketChannel.java:194)
at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:206) 
... 3 more

The master logs are

May 12, 2023 11:34:33 PM INFO jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/<ip>:50000 remote=<served>]] / Computer.threadPoolForRemoting [#903] for vsb-5 terminated: java.nio.channels.ClosedChannelException
May 12, 2023 11:41:15 PM INFO jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/<ip>:50000 remote=<server>]] / Computer.threadPoolForRemoting [#908] for vsb-4 terminated: java.nio.channels.ClosedChannelException
May 12, 2023 11:43:11 PM INFO hudson.slaves.ChannelPinger$1 onDead
Ping failed. Terminating the channel vsb-3.
java.util.concurrent.TimeoutException: Ping started at 1683927551978 hasn't completed by 1683927791979
	at hudson.remoting.PingThread.ping(PingThread.java:132)
	at hudson.remoting.PingThread.run(PingThread.java:88)

The UI node logs after disconnecting are

Connection was broken

java.nio.channels.ClosedChannelException
	at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:155)
	at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:180)
	at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:789)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:30)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:70)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

I checked everything related to the network, and I found that the port is listening on the host but is not listening inside the container.
The server metrics (CPU, RAM) are also good.

2

Just to clarify, you had this working before an update? If so, what version of Jenkins were you using previously that worked? Have you verified that the container 50000 is being redirected internally to the correct port in the container? Does Jenkins have port 50000 setup as the port to accept agent connections. Can you share your agent command line (with secrets redacted)?

3

docker-compose ports section:

 ports:
      - "8080:8080"
      - "50000:50000"

Agent command:

java.exe -jar pathToJar\agent.jar -jnlpUrl <server_ip>/computer/<my-agent>/slave-agent.jnlp -secret <secret> -workDir <workDir>

Note:
I haven’t updated Jenkins yet, everything was fine until two weeks ago, What is weird to me is that some other agents are still connected to the master and working, but all of a sudden some agents disconnect.
A master restart makes all agents work but after some time a random couple of them are disconnected, and they can’t reconnect.

4

Is the agent.jar file the latest from the controller (no longer called master)? Can you try downloading the latest from your controller and try that? Is there anything that changed 2 weeks ago?

5

Yes, It’s the latest version and the dashboard is not showing any warnings related to agent.jar version
.
Nothing changed, absolutely nothing.

6

You are using JDK 11 or 17 to run the controller and agents?

7

I’m using JDK 11 for both of them.

8

I don’t know what else to look at, sorry. We’ll have to see if someone else can help.