Jenkins setup:
Unfortunately, I am unable to retrieve my full Jenkins setup info since Jenkins does not start.
Jenkins version: 2.479.1
Linux version: Ubuntu 22.04.5 LTS
Java version: 21.0.5
Our Jenkins service is crashing on startup after upgrading to 2.479.1. Here is the exception from the log:
2024-11-19 20:07:34.189+0000 [id=77] WARNING o.j.p.w.cps.CpsFlowExecution$3#onFailure: could not resume org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3@769d14a6
org.jenkinsci.remoting.util.ExecutorServiceUtils$FatalRejectedExecutionException: Cannot execute the command java.util.concurrent.FutureTask@45811bbb[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@444d1502[Wrapped task = org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$$Lambda/0x0000765ae4a652f8@79c244c1]]. The executor service is shutting down
at hudson.remoting.SingleLaneExecutorService.execute(SingleLaneExecutorService.java:115)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123)
at jenkins.util.InterceptingExecutorService.submit(InterceptingExecutorService.java:46)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4.onSuccess(CpsFlowExecution.java:992)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4.onSuccess(CpsFlowExecution.java:987)
at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.concurrent.Futures$1.run(Futures.java:147)
at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.concurrent.DirectExecutor.execute(DirectExecutor.java:33)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1299)
at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:763)
at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:134)
at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:157)
at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:97)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.runInCpsVmThread(CpsFlowExecution.java:987)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.afterStepExecutionsResumed(CpsFlowExecution.java:947)
at PluginClassLoader for workflow-api//org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ParallelResumer.run(FlowExecutionList.java:402)
at PluginClassLoader for workflow-api//org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ParallelResumer.lambda$run$6(FlowExecutionList.java:433)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Here is the contents of the log file immediately before the exception:
2024-11-20 17:30:57.535+0000 [id=1] INFO winstone.Logger#logInternal: Beginning extraction from war file
2024-11-20 17:30:57.812+0000 [id=1] WARNING o.e.j.ee9.nested.ContextHandler#setContextPath: Empty contextPath
2024-11-20 17:30:58.088+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: jetty-12.0.13; built: 2024-09-03T03:04:05.240Z; git: 816018a420329c1cacd4116799cda8c8c60a57cd; jvm 21.0.5+11-Ubuntu-1ubuntu122.04
2024-11-20 17:31:00.072+0000 [id=1] INFO o.e.j.e.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.ee9.jsp.JettyJspServlet
2024-11-20 17:31:00.296+0000 [id=1] INFO o.e.j.s.DefaultSessionIdManager#doStart: Session workerName=node0
2024-11-20 17:31:01.750+0000 [id=1] INFO hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/lib/jenkins found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
2024-11-20 17:31:01.989+0000 [id=1] INFO o.e.j.s.handler.ContextHandler#doStart: Started oeje9n.ContextHandler$CoreContextHandler@3c1e3314{Jenkins v2.479.1,/,b=file:///var/cache/jenkins/war/,a=AVAILABLE,h=oeje9n.ContextHandler$CoreContextHandler$CoreToNestedHandler@4b770e40{STARTED}}
2024-11-20 17:31:02.015+0000 [id=1] INFO o.e.j.server.AbstractConnector#doStart: Started ServerConnector@3be8821f{HTTP/1.1, (http/1.1)}{0.0.0.0:80}
2024-11-20 17:31:02.062+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: Started oejs.Server@636be97c{STARTING}[12.0.13,sto=0] @7251ms
2024-11-20 17:31:02.089+0000 [id=31] INFO winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled
2024-11-20 17:31:02.937+0000 [id=37] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization
2024-11-20 17:31:03.141+0000 [id=39] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins
2024-11-20 17:31:12.560+0000 [id=37] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
2024-11-20 17:31:13.077+0000 [id=36] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins
2024-11-20 17:31:13.366+0000 [id=38] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
2024-11-20 17:31:15.697+0000 [id=37] INFO h.p.b.g.GlobalTimeOutConfiguration#load: global timeout not set
2024-11-20 17:31:17.797+0000 [id=37] INFO hudson.slaves.SlaveComputer#tryReconnect: Attempting to reconnect Windows Node 1
2024-11-20 17:31:17.840+0000 [id=39] INFO jenkins.InitReactorRunner$1#onAttained: System config loaded
2024-11-20 17:31:19.220+0000 [id=38] INFO jenkins.InitReactorRunner$1#onAttained: System config adapted
2024-11-20 17:31:19.334+0000 [id=36] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
2024-11-20 17:31:19.344+0000 [id=38] INFO o.j.p.l.NodesMirror#createNodeResources: lockable-resources-plugin: configure node resources
2024-11-20 17:31:19.356+0000 [id=36] INFO jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
2024-11-20 17:31:19.735+0000 [id=38] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization
2024-11-20 17:32:07.229+0000 [id=111] INFO h.triggers.SCMTrigger$Runner#run: SCM changes detected in DesignSystem. Triggering #385
2024-11-20 17:32:16.056+0000 [id=249] INFO o.j.p.l.queue.LockRunListener#onCompleted: DesignSystem #384
2024-11-20 17:32:16.135+0000 [id=249] INFO o.j.p.l.queue.LockRunListener#onDeleted: DesignSystem #355
2024-11-20 17:32:25.022+0000 [id=32] INFO winstone.Logger#logInternal: JVM is terminating. Shutting down Jetty
2024-11-20 17:32:25.026+0000 [id=32] INFO org.eclipse.jetty.server.Server#doStop: Stopped oejs.Server@636be97c{STOPPING}[12.0.13,sto=0]
2024-11-20 17:32:25.034+0000 [id=32] INFO o.e.j.server.AbstractConnector#doStop: Stopped ServerConnector@3be8821f{HTTP/1.1, (http/1.1)}{0.0.0.0:80}
2024-11-20 17:32:25.155+0000 [id=32] INFO hudson.lifecycle.Lifecycle#onStatusUpdate: Stopping Jenkins
2024-11-20 17:32:25.198+0000 [id=32] INFO jenkins.model.Jenkins$16#onAttained: Started termination
2024-11-20 17:32:25.250+0000 [id=32] INFO h.p.b.global.Lifecycle#shutdown: Shutdown complete - Global TimeOut ScheduledExecutorService had 0 tasks pending
2024-11-21 21:04:02.374+0000 [id=1] INFO winstone.Logger#logInternal: Beginning extraction from war file
2024-11-21 21:04:15.778+0000 [id=1] WARNING o.e.j.ee9.nested.ContextHandler#setContextPath: Empty contextPath
2024-11-21 21:04:15.986+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: jetty-12.0.13; built: 2024-09-03T03:04:05.240Z; git: 816018a420329c1cacd4116799cda8c8c60a57cd; jvm 21.0.5+11-Ubuntu-1ubuntu122.04
2024-11-21 21:04:20.275+0000 [id=1] INFO o.e.j.e.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.ee9.jsp.JettyJspServlet
2024-11-21 21:04:20.454+0000 [id=1] INFO o.e.j.s.DefaultSessionIdManager#doStart: Session workerName=node0
2024-11-21 21:04:22.605+0000 [id=1] INFO hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/lib/jenkins found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
2024-11-21 21:04:22.970+0000 [id=1] INFO o.e.j.s.handler.ContextHandler#doStart: Started oeje9n.ContextHandler$CoreContextHandler@2f1ea80d{Jenkins v2.479.1,/,b=file:///var/cache/jenkins/war/,a=AVAILABLE,h=oeje9n.ContextHandler$CoreContextHandler$CoreToNestedHandler@37191ef0{STARTED}}
2024-11-21 21:04:23.013+0000 [id=1] INFO o.e.j.server.AbstractConnector#doStart: Started ServerConnector@116a2108{HTTP/1.1, (http/1.1)}{0.0.0.0:80}
2024-11-21 21:04:23.075+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: Started oejs.Server@6f63b475{STARTING}[12.0.13,sto=0] @22426ms
2024-11-21 21:04:23.091+0000 [id=31] INFO winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled
2024-11-21 21:04:23.553+0000 [id=36] INFO hudson.PluginManager#loadDetachedPlugins: Upgrading Jenkins. The last running version was 2.462.3. This Jenkins is version 2.479.1.
2024-11-21 21:04:23.742+0000 [id=37] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization
2024-11-21 21:04:24.168+0000 [id=36] INFO hudson.PluginManager#loadDetachedPlugins: Upgraded Jenkins from version 2.462.3 to version 2.479.1. Loaded detached plugins (and dependencies): []
2024-11-21 21:04:25.268+0000 [id=36] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins
2024-11-21 21:04:39.942+0000 [id=41] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
2024-11-21 21:04:40.023+0000 [id=37] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins
2024-11-21 21:04:40.171+0000 [id=37] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
2024-11-21 21:04:41.549+0000 [id=40] INFO h.p.b.g.GlobalTimeOutConfiguration#load: global timeout not set
2024-11-21 21:04:43.686+0000 [id=41] INFO hudson.slaves.SlaveComputer#tryReconnect: Attempting to reconnect Windows Node 1
2024-11-21 21:04:43.724+0000 [id=41] INFO jenkins.InitReactorRunner$1#onAttained: System config loaded
2024-11-21 21:04:45.309+0000 [id=41] INFO jenkins.InitReactorRunner$1#onAttained: System config adapted
2024-11-21 21:04:45.464+0000 [id=37] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
2024-11-21 21:04:45.467+0000 [id=37] INFO o.j.p.l.NodesMirror#createNodeResources: lockable-resources-plugin: configure node resources
2024-11-21 21:04:45.472+0000 [id=36] INFO jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
2024-11-21 21:04:45.796+0000 [id=41] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization
2024-11-21 21:05:30.583+0000 [id=32] INFO winstone.Logger#logInternal: JVM is terminating. Shutting down Jetty
2024-11-21 21:05:30.597+0000 [id=32] INFO org.eclipse.jetty.server.Server#doStop: Stopped oejs.Server@6f63b475{STOPPING}[12.0.13,sto=0]
2024-11-21 21:05:30.605+0000 [id=32] INFO o.e.j.server.AbstractConnector#doStop: Stopped ServerConnector@116a2108{HTTP/1.1, (http/1.1)}{0.0.0.0:80}
2024-11-21 21:05:30.631+0000 [id=32] INFO hudson.lifecycle.Lifecycle#onStatusUpdate: Stopping Jenkins
2024-11-21 21:05:30.669+0000 [id=32] INFO jenkins.model.Jenkins$16#onAttained: Started termination
2024-11-21 21:05:30.675+0000 [id=32] INFO h.p.b.global.Lifecycle#shutdown: Shutdown complete - Global TimeOut ScheduledExecutorService had 0 tasks pending
2024-11-21 21:05:31.983+0000 [id=203] WARNING o.j.p.w.cps.CpsFlowExecution$3#onFailure: could not resume org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3@5b7b7703
org.jenkinsci.remoting.util.ExecutorServiceUtils$FatalRejectedExecutionException: Cannot execute the command java.util.concurrent.FutureTask@331167ec[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@5734f84c[Wrapped task = org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$$Lambda/0x000079d7ecb27a08@25a3ea0a]]. The executor service is shutting down
at hudson.remoting.SingleLaneExecutorService.execute(SingleLaneExecutorService.java:115)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123)
at jenkins.util.InterceptingExecutorService.submit(InterceptingExecutorService.java:46)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4.onSuccess(CpsFlowExecution.java:992)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4.onSuccess(CpsFlowExecution.java:987)
at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.concurrent.Futures$1.run(Futures.java:147)
at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.concurrent.DirectExecutor.execute(DirectExecutor.java:33)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1299)
at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:763)
at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:134)
at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:157)
at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:97)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.runInCpsVmThread(CpsFlowExecution.java:987)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.afterStepExecutionsResumed(CpsFlowExecution.java:947)
at PluginClassLoader for workflow-api//org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ParallelResumer.run(FlowExecutionList.java:402)
at PluginClassLoader for workflow-api//org.jenkinsci.plugins.workflow.flow.FlowExecutionList$ParallelResumer.lambda$run$6(FlowExecutionList.java:433)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
The system log shows that the SystemD daemon shuts down the Jenkins service shortly after starting it up due to a timeout. I don’t understand what is causing the timeout. Based on the Jenkins log, the Jenkins service is starting up normally. We are going to roll back some Linux OS updates that were installed recently to see if that’s what caused the issue.
There is a section in the systemd override on my Ubuntu 22.04 (from systemctl edit jenkins) that says:
#
# [Service]
# Type=notify
# NotifyAccess=main
# ExecStart=/usr/bin/jenkins
# Restart=on-failure
# SuccessExitStatus=143
#
# # Configures the time to wait for start-up. If Jenkins does not signal start-up
# # completion within the configured time, the service will be considered failed
# # and will be shut down again. Takes a unit-less value in seconds, or a time span
# # value such as "5min 20s". Pass "infinity" to disable the timeout logic.
# #TimeoutStartSec=90
You might try setting the TimeoutStartSec to a very large value.
@MarkEWaite - Thanks. I followed your suggestion above and that got me past the timeout. Once I had Jenkins running I figured out what caused the problem. The SSH connection to my Windows node had gone down, and I had around 160 jobs running and waiting for the node to come back online. That’s what was causing Jenkins to take so long to start up. I killed all the running jobs and all is well now.
Thank you for your assistance. You can close this ticket.
