My Jenkins startup times out. It appears to work, but the start command times out on the shell prompt

Hello all,

For the past few releases, I’ve had to manually intervene to shutdown and startup Jenkins. Startup would be my primary concern and, possibly, the cause of the shutdown problems so I’d like to know that it has started OK before I worry about not shutting it down cleanly.

I’ve just updated to 2.325 and it hasn’t made a difference.

The 2.325 startup output looks like:

$ date; sudo systemctl start jenkins
Thu 16 Dec 09:15:17 GMT 2021
Job for jenkins.service failed because a timeout was exceeded. See "systemctl status jenkins.service" and "journalctl -xe" for details.

$ sudo journalctl -xe
Dec 16 08:53:29 jenkinsserver systemd[1]: jenkins.service failed.
Dec 16 09:20:18 jenkinsserver systemd[1]: jenkins.service start operation timed out. Terminating.
Dec 16 09:20:18 jenkinsserver systemd[1]: Failed to start LSB: Jenkins Continuous Integration Server.
-- Subject: Unit jenkins.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit jenkins.service has failed.
--
-- The result is failed.
Dec 16 09:20:18 jenkinsserver systemd[1]: jenkins.service failed.

A subsequent “sudo systemctl stop jenkins” will appear to work, but leave the process running and the port busy.

Jenkins appears to generally operate OK, so it’s not an urgent problem. It is a little frustrating when trying to do maintenance, though.

The system log doesn’t give me much of a clue (portion below).

Any ideas what to poke about with, next?

Thanks,
Tim

2021-12-16 09:11:37.316+0000 [id=622]   INFO    hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Periodic background build discarder. 1,681 ms
Running from: /usr/lib/jenkins/jenkins.war
2021-12-16 09:15:18.956+0000 [id=1]     WARNING winstone.Logger#logInternal: Parameter handlerCountMax is now deprecated
2021-12-16 09:15:19.020+0000 [id=1]     WARNING winstone.Logger#logInternal: Parameter handlerCountMaxIdle is now deprecated
2021-12-16 09:15:19.069+0000 [id=1]     INFO    org.eclipse.jetty.util.log.Log#initialized: Logging initialized @1055ms to org.eclipse.jetty.util.log.JavaUtil
Log
2021-12-16 09:15:19.144+0000 [id=1]     INFO    winstone.Logger#logInternal: Beginning extraction from war file
2021-12-16 09:15:19.195+0000 [id=1]     WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath
2021-12-16 09:15:19.342+0000 [id=1]     INFO    org.eclipse.jetty.server.Server#doStart: jetty-9.4.43.v20210629; built: 2021-06-30T11:07:22.254Z; git: 526006e
cfa3af7f1a27ef3a288e2bef7ea9dd7e8; jvm 11.0.7+10-LTS
2021-12-16 09:15:19.974+0000 [id=1]     INFO    o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.Jet
tyJspServlet
2021-12-16 09:15:20.102+0000 [id=1]     INFO    o.e.j.s.s.DefaultSessionIdManager#doStart: DefaultSessionIdManager workerName=node0
2021-12-16 09:15:20.102+0000 [id=1]     INFO    o.e.j.s.s.DefaultSessionIdManager#doStart: No SessionScavenger set, using defaults
2021-12-16 09:15:20.104+0000 [id=1]     INFO    o.e.j.server.session.HouseKeeper#startScavenging: node0 Scavenging every 600000ms
2021-12-16 09:15:21.314+0000 [id=1]     INFO    hudson.WebAppMain#contextInitialized: Jenkins home directory: /jenkins/jenkins found at: SystemProperties.getP
roperty("JENKINS_HOME")
2021-12-16 09:15:21.756+0000 [id=1]     INFO    o.e.j.s.handler.ContextHandler#doStart: Started w.@11ce2e22{Jenkins v2.325,/,file:///var/cache/jenkins/war/,AV
AILABLE}{/var/cache/jenkins/war}
2021-12-16 09:15:21.796+0000 [id=1]     INFO    o.e.j.server.AbstractConnector#doStart: Started ServerConnector@6771beb3{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2021-12-16 09:15:21.796+0000 [id=1]     INFO    org.eclipse.jetty.server.Server#doStart: Started @3783ms
2021-12-16 09:15:21.824+0000 [id=23]    INFO    winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled
2021-12-16 09:15:22.680+0000 [id=30]    INFO    jenkins.InitReactorRunner$1#onAttained: Started initialization
2021-12-16 09:15:25.072+0000 [id=29]    INFO    jenkins.InitReactorRunner$1#onAttained: Listed all plugins
2021-12-16 09:15:32.517+0000 [id=29]    INFO    jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
2021-12-16 09:15:32.598+0000 [id=30]    INFO    jenkins.InitReactorRunner$1#onAttained: Started all plugins
2021-12-16 09:15:35.309+0000 [id=29]    INFO    jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
2021-12-16 09:15:35.679+0000 [id=30]    INFO    hudson.slaves.SlaveComputer#tryReconnect: Attempting to reconnect Dev 2019
2021-12-16 09:15:35.692+0000 [id=30]    INFO    hudson.slaves.SlaveComputer#tryReconnect: Attempting to reconnect Project Pack 2020
2021-12-16 09:15:35.734+0000 [id=30]    INFO    hudson.slaves.SlaveComputer#tryReconnect: Attempting to reconnect Solaris11
2021-12-16 09:15:35.735+0000 [id=30]    INFO    hudson.slaves.SlaveComputer#tryReconnect: Attempting to reconnect SuSE11
2021-12-16 09:15:35.797+0000 [id=30]    INFO    jenkins.InitReactorRunner$1#onAttained: System config loaded
2021-12-16 09:15:36.394+0000 [id=18]    INFO    o.j.p.s.StrictCrumbIssuer#validateCrumb: Invalid crumb found in the request
2021-12-16 09:15:36.418+0000 [id=11]    INFO    o.j.p.s.StrictCrumbIssuer#validateCrumb: Invalid crumb found in the request
2021-12-16 09:15:36.897+0000 [id=30]    INFO    jenkins.InitReactorRunner$1#onAttained: System config adapted
2021-12-16 09:15:37.591+0000 [id=60]    WARNING h.plugins.sshslaves.SSHLauncher#checkJavaIsInPath: Java is not in the PATH nor configured with the javaPath se
tting, Jenkins will try to guess where is Java, this guess will be removed in the future. :Launch agents via SSH
2021-12-16 09:15:38.115+0000 [id=29]    INFO    jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
2021-12-16 09:15:38.132+0000 [id=30]    INFO    jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
2021-12-16 09:15:38.323+0000 [id=111]   INFO    hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Download metadata
2021-12-16 09:15:38.347+0000 [id=111]   INFO    hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Download metadata. 24 ms
2021-12-16 09:15:38.774+0000 [id=30]    INFO    jenkins.InitReactorRunner$1#onAttained: Completed initialization
2021-12-16 09:15:39.234+0000 [id=22]    WARNING j.branch.WorkspaceLocatorImpl#getWorkspaceRoot: JENKINS-2111 path sanitization ineffective when using legacy W
orkspace Root Directory ‘${ITEM_ROOTDIR}/workspace’; switch to ‘${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}’ as in JENKINS-8446 / JENKINS-21942
2021-12-16 09:15:39.281+0000 [id=22]    INFO    hudson.WebAppMain$3#run: Jenkins is fully up and running
2021-12-16 09:15:41.262+0000 [id=153]   INFO    h.TcpSlaveAgentListener$ConnectionHandler#run: Connection #1 failed: java.io.EOFException
2021-12-16 09:15:41.288+0000 [id=154]   INFO    h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #2 from /10.23.135.73:55677
2021-12-16 09:15:41.310+0000 [id=155]   INFO    h.TcpSlaveAgentListener$ConnectionHandler#run: Connection #3 failed: java.io.EOFException
2021-12-16 09:15:41.389+0000 [id=159]   INFO    h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #4 from /10.23.135.16:62618
2021-12-16 09:15:44.542+0000 [id=59]    INFO    o.j.p.m.MailWatcherNotification#log: mail-watcher-plugin: notified: Computer Dev 2019 marked online
[12/16/21 09:15:44] SSH Launch of Dev 2019 on server1r1 completed in 8,453 ms
2021-12-16 09:15:44.592+0000 [id=58]    INFO    o.j.p.m.MailWatcherNotification#log: mail-watcher-plugin: notified: Computer SuSE11 marked online
2021-12-16 09:15:44.592+0000 [id=60]    INFO    o.j.p.m.MailWatcherNotification#log: mail-watcher-plugin: notified: Computer Project Pack 2020 marked online
[12/16/21 09:15:44] SSH Launch of SuSE11 on server1sh completed in 8,558 ms
[12/16/21 09:15:44] SSH Launch of Project Pack 2020 on server2r1 completed in 8,559 ms
2021-12-16 09:15:45.213+0000 [id=200]   INFO    h.TcpSlaveAgentListener$ConnectionHandler#run: Connection #5 failed: java.io.EOFException
2021-12-16 09:15:45.268+0000 [id=201]   INFO    h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #6 from /10.23.135.63:55200
2021-12-16 09:15:45.593+0000 [id=145]   INFO    o.j.p.m.MailWatcherNotification#log: mail-watcher-plugin: notified: Computer Win2019 marked online
2021-12-16 09:15:46.366+0000 [id=151]   INFO    o.j.p.m.MailWatcherNotification#log: mail-watcher-plugin: notified: Computer Win2016 marked online
2021-12-16 09:15:48.210+0000 [id=61]    INFO    o.j.p.m.MailWatcherNotification#log: mail-watcher-plugin: notified: Computer Solaris11 marked online
[12/16/21 09:15:48] SSH Launch of Solaris11 on server4r1 completed in 12,106 ms
2021-12-16 09:15:49.183+0000 [id=52]    INFO    o.j.p.m.MailWatcherNotification#log: mail-watcher-plugin: notified: Computer RedHat CI marked online
2021-12-16 09:37:24.136+0000 [id=341]   INFO    hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Periodic background build discarder

That looks like a startup log not a shutdown log (due to the fact it keeps saying startup).

My recommendation would be to increase the timeout - centos - How to change systemd service timeout value? - Unix & Linux Stack Exchange

My guess would be that mail watcher plugin, its over 5 years old, and I could easily see it having some hook somewhere that isn’t letting things shutdown. I don’t know though.

Thank you for the suggestion. I think I gave it long enough …

$ sudo systemctl show jenkins.service -p TimeoutStopUSec
TimeoutStopUSec=35min
$ date; time sudo systemctl start jenkins
Fri 17 Dec 09:08:25 GMT 2021
Job for jenkins.service failed because a timeout was exceeded. See "systemctl status jenkins.service" and "journalctl -xe" for details.

real    35m0.089s
user    0m0.015s
sys     0m0.018s