Hi, I have a Jenkins Server setup with 3 Jenkins Agents one physical Windows machines. They’re all communicating effectively and I’m running jobs on all 3 machines. The job consists of installing a software, running TestComplete tests, and uninstalling the software. I’m adding a reboot step to the job and this is where the trouble comes in.
Two of the agents are machines in our local office in Massachusetts and these two run fine, including the reboot step. I restart the machine using ‘bat"shutdown /r /f /t 0"’ and the pipeline outputs this line “Cannot contact QA-Auto-Temp-SUT-W02: hudson.remoting.ChannelClosedException: Channel “hudson.remoting.Channel@61a7cfa8:QA-Auto-Temp-SUT-W02”: Remote call on QA-Auto-Temp-SUT-W02 failed. The channel is closing down or has closed down”. The pipeline seems to wait while the agent is offline, but when the agent comes back online (agent command is run when the machine logs back in), it automatically resumes the pipeline. This is exactly as I want it, and it works perfectly.
The other agent is in our office in India, which is connected to our main office by IPSec. Every other part of the job works, with files being copied and software installed and TestComplete tests run, but when I introduced the reboot step there is a different behavior. Instead of “waiting” for the agent to come back online, the pipeline fails as soon as the agent reboots and prints a stack trace error (included below). Does anyone know why this particular reboot step works on a local machine but not one connected over IPSec, if that is the issue? Is it a network issue? Does anyone have a suggestion on how to make this work?
I’ll greatly appreciate any help on this. Thank you!
Stack trace error:
hudson.remoting.ProxyException: java.nio.channels.ClosedChannelException
at jenkins.agents.WebSocketAgents$Session.closed(WebSocketAgents.java:157)
Also: hudson.remoting.ProxyException: hudson.remoting.Channel$CallSiteStackTrace: Remote call to QA-Auto-Temp-SUT-P01
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
at hudson.remoting.Request.call(Request.java:199)
at hudson.remoting.Channel.call(Channel.java:999)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285)
at jdk.proxy2/jdk.proxy2.$Proxy120.getIOtriplet(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.(Launcher.java:1184)
at hudson.Launcher$RemoteLauncher.launch(Launcher.java:1124)
at hudson.Launcher$ProcStarter.start(Launcher.java:509)
at org.jenkinsci.plugins.durabletask.WindowsBatchScript.doLaunch(WindowsBatchScript.java:105)
at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launchWithCookie(FileMonitoringTask.java:141)
at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:134)
at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:329)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:322)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:196)
at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:124)
at jdk.internal.reflect.GeneratedMethodAccessor16093.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1225)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:41)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:180)
at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:163)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:178)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:182)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:152)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:152)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:152)
at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:90)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:116)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:85)
at jdk.internal.reflect.GeneratedMethodAccessor151.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
at com.cloudbees.groovy.cps.Next.step(Next.java:83)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:152)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:146)
at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:146)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:187)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:422)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:330)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:294)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
Also: org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: c2538028-9966-49d5-b01a-4c1ee20d4238
Caused: hudson.remoting.ProxyException: hudson.remoting.RequestAbortedException: java.nio.channels.ClosedChannelException
at hudson.remoting.Request.abort(Request.java:346)
at hudson.remoting.Channel.terminate(Channel.java:1080)
at hudson.remoting.Channel$1.terminate(Channel.java:620)
at hudson.remoting.AbstractByteBufferCommandTransport.terminate(AbstractByteBufferCommandTransport.java:356)
at jenkins.agents.WebSocketAgents$Session.closed(WebSocketAgents.java:158)
at jenkins.websocket.WebSockets$1.onWebSocketClose(WebSockets.java:88)
at jenkins.websocket.WebSockets$1.onWebSocketError(WebSockets.java:94)
at jenkins.websocket.Jetty10Provider$2.onWebSocketError(Jetty10Provider.java:174)
at org.eclipse.jetty.websocket.common.JettyWebSocketFrameHandler.onError(JettyWebSocketFrameHandler.java:260)
at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.lambda$closeConnection$2(WebSocketCoreSession.java:284)
at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1468)
at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1487)
at org.eclipse.jetty.websocket.core.server.internal.AbstractHandshaker$1.handle(AbstractHandshaker.java:212)
at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.closeConnection(WebSocketCoreSession.java:284)
at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.onEof(WebSocketCoreSession.java:254)
at org.eclipse.jetty.websocket.core.internal.WebSocketConnection.fillAndParse(WebSocketConnection.java:482)
at org.eclipse.jetty.websocket.core.internal.WebSocketConnection.onFillable(WebSocketConnection.java:340)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
at java.base/java.lang.Thread.run(Thread.java:833)
Jenkins setup:
Jenkins: 2.401.3
OS: Windows Server 2012 R2 - 6.3
Java: 17.0.8 - Oracle Corporation (Java HotSpot™ 64-Bit Server VM)
TestComplete:2.9.2
ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:159.v25b_c67cd35fb_
apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5
bootstrap5-api:5.3.0-1
bouncycastle-api:2.29
branch-api:2.1122.v09cb_8ea_8a_724
build-timeout:1.31
caffeine-api:3.1.6-115.vb_8b_b_328e59d8
checks-api:2.0.0
cloudbees-folder:6.815.v0dd5a_cb_40e0e
commons-lang3-api:3.12.0-36.vd97de6465d5b_
commons-text-api:1.10.0-36.vc008c8fcda_7b_
credentials:1271.v54b_1c2c6388a_
credentials-binding:631.v861c06d062b_4
display-url-api:2.3.7
durable-task:513.vc48a_a_075a_d93
echarts-api:5.4.0-5
email-ext:2.100
font-awesome-api:6.4.0-2
git:5.2.0
git-client:4.4.0
github:1.37.1
github-api:1.314-431.v78d72a_3fe4c3
github-branch-source:1728.v859147241f49
gradle:2.8.2
instance-identity:173.va_37c494ec4e5
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.15.2-350.v0c2f3f8fc595
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.8-1
jjwt-api:0.11.5-77.v646c772fddb_0
job-dsl:1.84
jquery3-api:3.7.0-1
junit:1217.v4297208a_a_b_ce
ldap:682.v7b_544c9d1512
mailer:457.v3f72cb_e015e5
matrix-auth:3.1.10
matrix-project:789.v57a_725b_63c79
mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_
mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_
nodelabelparameter:1.12.0
okhttp-api:4.11.0-145.vcb_8de402ef81
pam-auth:1.10
pipeline-build-step:505.v5f0844d8d126
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:656.va_a_ceeb_6ffb_f7
pipeline-input-step:468.va_5db_051498a_4
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2144.v077a_d1928a_40
pipeline-model-definition:2.2144.v077a_d1928a_40
pipeline-model-extensions:2.2144.v077a_d1928a_40
pipeline-rest-api:2.33
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2144.v077a_d1928a_40
pipeline-stage-view:2.33
pipeline-utility-steps:2.16.0
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:3.3.0
resource-disposer:0.22
schedule-build:496.v10a_2b_7da_5600
scm-api:676.v886669a_199a_a_
script-security:1251.vfe552ed55f8d
snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4
ssh-credentials:305.v8f4381501156
ssh-slaves:2.877.v365f5eb_a_b_eec
sshd:3.303.vefc7119b_ec23
structs:324.va_f5d6774f3a_d
timestamper:1.25
token-macro:359.vb_cde11682e0c
trilead-api:2.84.v72119de229b_7
variant:59.vf075fe829ccb
workflow-aggregator:596.v8c21c963d92d
workflow-api:1251.vd4889a_b_0a_065
workflow-basic-steps:1017.vb_45b_302f0cea_
workflow-cps:3731.ve4b_5b_857b_a_d3
workflow-durable-task-step:1278.v94b_dc2b_50c6f
workflow-job:1316.vd2290d3341a_f
workflow-multibranch:756.v891d88f2cd46
workflow-scm-step:415.v434365564324
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:848.v5a_383b_d14921
ws-cleanup:0.45