# Jenkins agents randomly disconnecting from controller

**URL:** <https://community.jenkins.io/t/jenkins-agents-randomly-disconnecting-from-controller/2112>\
**Category:** Ask a question\
**Tags:** question\
**Created:** [April 14, 2022, 10:01pm UTC](https://community.jenkins.io/t/jenkins-agents-randomly-disconnecting-from-controller/2112 "2022-04-14T22:01:34Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![johnny\_m](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/johnny_m/32/1130_2.png) [@johnny\_m](https://community.jenkins.io/u/johnny_m)\
**Post date:** [April 14, 2022, 10:01pm UTC](https://community.jenkins.io/t/jenkins-agents-randomly-disconnecting-from-controller/2112/1 "2022-04-14T22:01:34Z")

</div>

Hello all,  
I’m hoping someone here can give me direction on how to fix random disconnects from build agents in my Jenkins environment. The past few months I have been experiencing agents intermittently disconnecting from the Jenkins controller. I have tried various fixes such as tuning the ssh config on the controller and agents, ensuring I have the same Java version on the controller along with the agents. I have rebuilt the controller and both agents, however I’m still experiencing random disconnects when building docker images. I have listed below the environment configuration along with some logs that I’m seeing when these agents disconnect.

I was not able to attach logs w/ the support plugin… but, do have the information if other logs need to be provided. I super appreciate any direction… 🙂

**Environment and configuration**  
Jenkins Core Version: 2.319.2  
Jenkins Controller OS: Amazon Linux II  
Jenkins Agents: Ubuntu 20.04  
Java Version:  
Controller - openjdk version “1.8.0\_312”  
Build Agent (ARM Instance) - openjdk version “1.8.0\_312”  
Build Agent (x64\_x86 instance) -  
OpenSSH Version: Controller: OpenSSH\_7.4p1 ---- Build Agents: OpenSSH\_8.2p1

**Agent Configs:**

```auto
<slave>
<name>ARM</name>
<description>ARM Build Agent - Ubuntu 20.04</description>
<remoteFS>/home/ubuntu</remoteFS>
<numExecutors>6</numExecutors>
<mode>NORMAL</mode>
<retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
<launcher class="hudson.plugins.sshslaves.SSHLauncher" plugin="ssh-slaves@1.33.0">
<host>REDACTED</host>
<port>22</port>
<credentialsId>build-agent-ed25519</credentialsId>
<launchTimeoutSeconds>60</launchTimeoutSeconds>
<maxNumRetries>10</maxNumRetries>
<retryWaitTime>15</retryWaitTime>
<sshHostKeyVerificationStrategy class="hudson.plugins.sshslaves.verifiers.KnownHostsFileKeyVerificationStrategy"/>
<tcpNoDelay>true</tcpNoDelay>
</launcher>
<label>ARM</label>
<nodeProperties>
<hudson.slaves.EnvironmentVariablesNodeProperty>
<envVars serialization="custom">
<unserializable-parents/>
<tree-map>
<default>
<comparator class="java.lang.String$CaseInsensitiveComparator"/>
</default>
<int>1</int>
<string>Java</string>
<string>/usr/lib/jvm/java-8-openjdk-arm64</string>
</tree-map>
</envVars>
</hudson.slaves.EnvironmentVariablesNodeProperty>
<hudson.tools.ToolLocationNodeProperty>
<locations>
<hudson.tools.ToolLocationNodeProperty_-ToolLocation>
<type>hudson.plugins.git.GitTool$DescriptorImpl</type>
<name>git</name>
<home>/usr/bin/git</home>
</hudson.tools.ToolLocationNodeProperty_-ToolLocation>
</locations>
</hudson.tools.ToolLocationNodeProperty>
</nodeProperties>
</slave>

```

**Second Agent**

```auto
<slave>
<name>x86</name>
<description>x86 Build Server</description>
<remoteFS>/home/ubuntu</remoteFS>
<numExecutors>4</numExecutors>
<mode>NORMAL</mode>
<retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
<launcher class="hudson.plugins.sshslaves.SSHLauncher" plugin="ssh-slaves@1.33.0">
<host>REDACTED</host>
<port>22</port>
<credentialsId>build-agent-ed25519</credentialsId>
<launchTimeoutSeconds>60</launchTimeoutSeconds>
<maxNumRetries>10</maxNumRetries>
<retryWaitTime>15</retryWaitTime>
<sshHostKeyVerificationStrategy class="hudson.plugins.sshslaves.verifiers.KnownHostsFileKeyVerificationStrategy"/>
<tcpNoDelay>true</tcpNoDelay>
</launcher>
<label>x86</label>
<nodeProperties>
<hudson.slaves.EnvironmentVariablesNodeProperty>
<envVars serialization="custom">
<unserializable-parents/>
<tree-map>
<default>
<comparator class="java.lang.String$CaseInsensitiveComparator"/>
</default>
<int>1</int>
<string>Java</string>
<string>/usr/lib/jvm/java-1.8.0-openjdk-amd64</string>
</tree-map>
</envVars>
</hudson.slaves.EnvironmentVariablesNodeProperty>
<hudson.tools.ToolLocationNodeProperty>
<locations>
<hudson.tools.ToolLocationNodeProperty_-ToolLocation>
<type>hudson.plugins.git.GitTool$DescriptorImpl</type>
<name>git</name>
<home>/usr/bin/git</home>
</hudson.tools.ToolLocationNodeProperty_-ToolLocation>
</locations>
</hudson.tools.ToolLocationNodeProperty>
</nodeProperties>
</slave>

```

**Agent Log when experiencing connectivity problems**

```auto
Apr 05, 2022 3:35:03 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
INFO: I/O error in channel channel
java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:75)
Caused by: java.io.EOFException
	at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2905)
	at java.base/java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3400)
	at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:936)
	at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379)
	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
	at hudson.remoting.Command.readFrom(Command.java:142)
	at hudson.remoting.Command.readFrom(Command.java:128)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)

```

**Another Agent Log When Problem is Present**

```auto
The kexTimeout (65000 ms) expired.
SSH Connection failed with IOException: "The kexTimeout (65000 ms) expired.", retrying in 15 seconds. There are 10 more retries left.
The kexTimeout (65000 ms) expired.
SSH Connection failed with IOException: "The kexTimeout (65000 ms) expired.", retrying in 15 seconds. There are 9 more retries left.
The kexTimeout (65000 ms) expired.
SSH Connection failed with IOException: "The kexTimeout (65000 ms) expired.", retrying in 15 seconds. There are 8 more retries left.
The kexTimeout (65000 ms) expired.
SSH Connection failed with IOException: "The kexTimeout (65000 ms) expired.", retrying in 15 seconds. There are 7 more retries left.
connect timed out

```

---

<div class="post-metadata">

**Author:** ![halkeye](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/halkeye/32/8_2.png) [@halkeye](https://community.jenkins.io/u/halkeye)\
**Post date:** [April 15, 2022, 7:49am UTC](https://community.jenkins.io/t/jenkins-agents-randomly-disconnecting-from-controller/2112/2 "2022-04-15T07:49:20Z")

</div>

> [@johnny\_m](#):
>
> `The kexTimeout (65000 ms) expired.`

I don’t know what kex timeout is but it feels like the problem.

You could try using jnlp and having your agents connect to your controller instead. That should eliminate ssh related issues?

---

<div class="post-metadata">

**Author:** ![johnny\_m](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/johnny_m/32/1130_2.png) [@johnny\_m](https://community.jenkins.io/u/johnny_m)\
**Post date:** [April 15, 2022, 3:07pm UTC](https://community.jenkins.io/t/jenkins-agents-randomly-disconnecting-from-controller/2112/3 "2022-04-15T15:07:05Z")

</div>

Hey Halkeye… 🙂 Thanks for the response… I was thinking about trying jnlp… I’ll give that a shot tonight after hours…

Thanks again.

Johnny M

---

<div class="post-metadata">

**Author:** ![saper](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/saper/32/138_2.png) [@saper](https://community.jenkins.io/u/saper)\
**Post date:** [April 20, 2022, 1:11pm UTC](https://community.jenkins.io/t/jenkins-agents-randomly-disconnecting-from-controller/2112/4 "2022-04-20T13:11:25Z")

</div>

This can be related to your AWS configuration:

> <https://stackoverflow.com/questions/41965115/unable-to-connect-to-aws-slave-via-jenkins/42007462#42007462>

---

<div class="post-metadata">

**Author:** ![johnny\_m](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/johnny_m/32/1130_2.png) [@johnny\_m](https://community.jenkins.io/u/johnny_m)\
**Post date:** [April 20, 2022, 3:00pm UTC](https://community.jenkins.io/t/jenkins-agents-randomly-disconnecting-from-controller/2112/5 "2022-04-20T15:00:16Z")

</div>

Hey Saper,  
I wanted to reach out and say thank you for your response… Currently all traffic is is allowed from the agent IP to the controller and all traffic from the controller to the agent is allowed.

I also have tried using JNLP to connect from the agent to the controller and was able to establish a connection, but it appears to be unreliable as well… Whenever I close the shell to one of my build servers the JNLP connection drops.

Im wondering if you guys may have any other suggestions??? Ive been banging on my head on this one for a while now… Its been somewhat difficult to troubleshoot with the problem being intermittent.

Thanks again.

---

<div class="post-metadata">

**Author:** ![saper](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/saper/32/138_2.png) [@saper](https://community.jenkins.io/u/saper)\
**Post date:** [April 21, 2022, 12:32pm UTC](https://community.jenkins.io/t/jenkins-agents-randomly-disconnecting-from-controller/2112/6 "2022-04-21T12:32:49Z")

</div>

> [@johnny\_m](#):
>
> Whenever I close the shell to one of my build servers the JNLP connection drops.

This is interesting, are you using SSH tunnel to have the connection open?

I am afraid your problem is not related to Jenkins at all - you have a networking problem between the controller and the agent. My way to troubleshoot this would be to:

- Understand if there is a middlebox - a firewall / NAT / whatever between the agent and the controller - if yes, examine the settings such as NAT timeout or similar.
- Sniff the traffic using tools like `tcpdump` on 1) controller 2) agent and 3) middlebox and see which packets are lost and where

---

<div class="post-metadata">

**Author:** ![johnny\_m](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/johnny_m/32/1130_2.png) [@johnny\_m](https://community.jenkins.io/u/johnny_m)\
**Post date:** [April 29, 2022, 3:56pm UTC](https://community.jenkins.io/t/jenkins-agents-randomly-disconnecting-from-controller/2112/7 "2022-04-29T15:56:18Z")

</div>

Hey Saper,  
Thank you for the post… My environment is hosted in AWS and generally speaking the disconnects weren’t persistent before an upgrade in January of 2022. I tried using JNLP agents as well, but… still experiencing the Intermittent disconnects… from only one of the two build agents.

The latest logs on the agent disconnecting are listed below… Im really hoping I can find a fix to this… I think I will take your advice and run a tcpdump on the agent and controller. Thanks again,

```auto
Apr 27, 2022 11:07:55 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request UserRequest:hudson.FilePath$IsDirectory@77a0621
java.io.IOException: Broken pipe
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:313)
	at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:84)
	at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:92)
	at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:65)
	at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
	at hudson.remoting.Channel.send(Channel.java:766)
	at hudson.remoting.Request$2.run(Request.java:389)
	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Apr 27, 2022 11:07:55 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
INFO: I/O error in channel channel
java.io.IOException: Unexpected EOF
	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:100)
	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)

```

---

<div class="post-metadata">

**Author:** ![johnny\_m](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/johnny_m/32/1130_2.png) [@johnny\_m](https://community.jenkins.io/u/johnny_m)\
**Post date:** [April 29, 2022, 3:57pm UTC](https://community.jenkins.io/t/jenkins-agents-randomly-disconnecting-from-controller/2112/8 "2022-04-29T15:57:24Z")

</div>

```auto
Apr 27, 2022 11:07:55 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request UserRequest:hudson.FilePath$IsDirectory@51a4c24e
java.io.IOException: Broken pipe
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:313)
	at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:84)
	at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:92)
	at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:88)
	at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:57)
	at java.io.OutputStream.write(OutputStream.java:75)
	at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
	at hudson.remoting.Channel.send(Channel.java:766)
	at hudson.remoting.Request$2.run(Request.java:389)
	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Apr 27, 2022 10:23:13 PM hudson.remoting.RemoteInvocationHandler$Unexporter run
WARNING: Couldn't clean up oid=2 from null
java.io.IOException: Broken pipe
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:313)
	at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:84)
	at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:92)
	at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:65)
	at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
	at hudson.remoting.Channel.send(Channel.java:766)
	at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.cleanup(RemoteInvocationHandler.java:398)
	at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.access$1000(RemoteInvocationHandler.java:357)
	at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:615)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:121)
	at java.lang.Thread.run(Thread.java:748)

```