I’m currently using:
Linux 3.10.0-1160.92.1.el7.x86_64 x86_64
Jenkins 2.426.1
Running on a kubernetes environment. The nodes are virtualized windows machines most of them windows 11.
What happens is that during some builds (freestyle projects) the communication channel with the controller breaks and the build is aborted.
This is something we were would see occasionally but lately it have become much more frequent, impacting several nodes in the same day. It appears to be some networking issue, although through ping we have never seen any loss of connectivity at the time these issues happened. The stack is shown at the end of the post.
We have tried tried disabling the ping from the nodes: (-Dhudson.slaves.ChannelPinger.pingTimeoutSeconds=-1, -Dhudson.slaves.ChannelPinger.pingIntervalSeconds=-1) but it didn’t help.
I have seen JIRAs still open for this issue so it appears to be a jenkins bug. But does anyone know of a workaround to prevent this timeout from happening?
Here are the logs:
java.nio.channels.ClosedChannelException
at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:241)
at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:221)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:825)
at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:289)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:177)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:279)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:501)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:244)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:196)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:209)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:793)
at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:172)
at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:343)
at hudson.remoting.Channel.close(Channel.java:1494)
at hudson.remoting.Channel.close(Channel.java:1447)
at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:939)
at hudson.slaves.SlaveComputer.kill(SlaveComputer.java:905)
at hudson.model.AbstractCIBase.killComputer(AbstractCIBase.java:97)
at jenkins.model.Jenkins.lambda$_cleanUpDisconnectComputers$11(Jenkins.java:3751)
at hudson.model.Queue._withLock(Queue.java:1397)
at hudson.model.Queue.withLock(Queue.java:1271)
at jenkins.model.Jenkins._cleanUpDisconnectComputers(Jenkins.java:3747)
at jenkins.model.Jenkins.cleanUp(Jenkins.java:3628)
at hudson.WebAppMain.contextDestroyed(WebAppMain.java:374)
at org.eclipse.jetty.server.handler.ContextHandler.callContextDestroyed(ContextHandler.java:1059)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextDestroyed(ServletContextHandler.java:636)
at org.eclipse.jetty.server.handler.ContextHandler.contextDestroyed(ContextHandler.java:1016)
at org.eclipse.jetty.servlet.ServletHandler.doStop(ServletHandler.java:306)
at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:97)
at org.eclipse.jetty.security.SecurityHandler.doStop(SecurityHandler.java:411)
at org.eclipse.jetty.security.ConstraintSecurityHandler.doStop(ConstraintSecurityHandler.java:413)
at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:97)
at org.eclipse.jetty.server.session.SessionHandler.doStop(SessionHandler.java:497)
at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:97)
at org.eclipse.jetty.server.handler.ContextHandler.stopContext(ContextHandler.java:1039)
at org.eclipse.jetty.servlet.ServletContextHandler.stopContext(ServletContextHandler.java:399)
at org.eclipse.jetty.webapp.WebAppContext.stopContext(WebAppContext.java:1311)
at org.eclipse.jetty.server.handler.ContextHandler.doStop(ContextHandler.java:1087)
at org.eclipse.jetty.servlet.ServletContextHandler.doStop(ServletContextHandler.java:312)
at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStop(Server.java:517)
at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
at winstone.Launcher.shutdown(Launcher.java:442)
at winstone.ShutdownHook.run(ShutdownHook.java:28)
Caused: java.io.IOException: Backing channel 'JNLP4-connect connection from **************' is disconnected.
at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:215)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285)
at com.sun.proxy.$Proxy92.isAlive(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1212)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1204)
at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:195)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:145)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:818)
at hudson.model.Build$BuildExecution.build(Build.java:199)
at hudson.model.Build$BuildExecution.doRun(Build.java:164)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:526)
at hudson.model.Run.execute(Run.java:1900)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)
FATAL: Unable to delete script file C:\Users\CRUISE~1\AppData\Local\Temp\jenkins5205634626611760213.bat
java.nio.channels.ClosedChannelException
at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:241)
at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:221)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:825)
at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:289)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:177)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:279)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:501)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:244)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:196)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:209)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:793)
at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:172)
at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:343)
at hudson.remoting.Channel.close(Channel.java:1494)
at hudson.remoting.Channel.close(Channel.java:1447)
at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:939)
at hudson.slaves.SlaveComputer.kill(SlaveComputer.java:905)
at hudson.model.AbstractCIBase.killComputer(AbstractCIBase.java:97)
at jenkins.model.Jenkins.lambda$_cleanUpDisconnectComputers$11(Jenkins.java:3751)
at hudson.model.Queue._withLock(Queue.java:1397)
at hudson.model.Queue.withLock(Queue.java:1271)
at jenkins.model.Jenkins._cleanUpDisconnectComputers(Jenkins.java:3747)
at jenkins.model.Jenkins.cleanUp(Jenkins.java:3628)
at hudson.WebAppMain.contextDestroyed(WebAppMain.java:374)
at org.eclipse.jetty.server.handler.ContextHandler.callContextDestroyed(ContextHandler.java:1059)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextDestroyed(ServletContextHandler.java:636)
at org.eclipse.jetty.server.handler.ContextHandler.contextDestroyed(ContextHandler.java:1016)
at org.eclipse.jetty.servlet.ServletHandler.doStop(ServletHandler.java:306)
at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:97)
at org.eclipse.jetty.security.SecurityHandler.doStop(SecurityHandler.java:411)
at org.eclipse.jetty.security.ConstraintSecurityHandler.doStop(ConstraintSecurityHandler.java:413)
at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:97)
at org.eclipse.jetty.server.session.SessionHandler.doStop(SessionHandler.java:497)
at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:97)
at org.eclipse.jetty.server.handler.ContextHandler.stopContext(ContextHandler.java:1039)
at org.eclipse.jetty.servlet.ServletContextHandler.stopContext(ServletContextHandler.java:399)
at org.eclipse.jetty.webapp.WebAppContext.stopContext(WebAppContext.java:1311)
at org.eclipse.jetty.server.handler.ContextHandler.doStop(ContextHandler.java:1087)
at org.eclipse.jetty.servlet.ServletContextHandler.doStop(ServletContextHandler.java:312)
at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStop(Server.java:517)
at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
at winstone.Launcher.shutdown(Launcher.java:442)
at winstone.ShutdownHook.run(ShutdownHook.java:28)
Caused: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@7716a117:JNLP4-connect connection from **************": Remote call on JNLP4-connect connection from ************** failed. The channel is closing down or has closed down
at hudson.remoting.Channel.call(Channel.java:993)
at hudson.FilePath.act(FilePath.java:1192)
at hudson.FilePath.act(FilePath.java:1181)
at hudson.FilePath.delete(FilePath.java:1728)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:163)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:818)
at hudson.model.Build$BuildExecution.build(Build.java:199)
at hudson.model.Build$BuildExecution.doRun(Build.java:164)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:526)
at hudson.model.Run.execute(Run.java:1900)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)
Build step 'Execute Windows batch command' marked build as failure