My Jenkins startup times out. It appears to work, but the start command times out on the shell prompt

Using Jenkins
1

Hello all,

For the past few releases, I’ve had to manually intervene to shutdown and startup Jenkins. Startup would be my primary concern and, possibly, the cause of the shutdown problems so I’d like to know that it has started OK before I worry about not shutting it down cleanly.

I’ve just updated to 2.325 and it hasn’t made a difference.

The 2.325 startup output looks like:

$ date; sudo systemctl start jenkins
Thu 16 Dec 09:15:17 GMT 2021
Job for jenkins.service failed because a timeout was exceeded. See "systemctl status jenkins.service" and "journalctl -xe" for details.

$ sudo journalctl -xe
Dec 16 08:53:29 jenkinsserver systemd[1]: jenkins.service failed.
Dec 16 09:20:18 jenkinsserver systemd[1]: jenkins.service start operation timed out. Terminating.
Dec 16 09:20:18 jenkinsserver systemd[1]: Failed to start LSB: Jenkins Continuous Integration Server.
-- Subject: Unit jenkins.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit jenkins.service has failed.
--
-- The result is failed.
Dec 16 09:20:18 jenkinsserver systemd[1]: jenkins.service failed.

A subsequent “sudo systemctl stop jenkins” will appear to work, but leave the process running and the port busy.

Jenkins appears to generally operate OK, so it’s not an urgent problem. It is a little frustrating when trying to do maintenance, though.

The system log doesn’t give me much of a clue (portion below).

Any ideas what to poke about with, next?

Thanks,
Tim

2021-12-16 09:11:37.316+0000 [id=622]   INFO    hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Periodic background build discarder. 1,681 ms
Running from: /usr/lib/jenkins/jenkins.war
2021-12-16 09:15:18.956+0000 [id=1]     WARNING winstone.Logger#logInternal: Parameter handlerCountMax is now deprecated
2021-12-16 09:15:19.020+0000 [id=1]     WARNING winstone.Logger#logInternal: Parameter handlerCountMaxIdle is now deprecated
2021-12-16 09:15:19.069+0000 [id=1]     INFO    org.eclipse.jetty.util.log.Log#initialized: Logging initialized @1055ms to org.eclipse.jetty.util.log.JavaUtil
Log
2021-12-16 09:15:19.144+0000 [id=1]     INFO    winstone.Logger#logInternal: Beginning extraction from war file
2021-12-16 09:15:19.195+0000 [id=1]     WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath
2021-12-16 09:15:19.342+0000 [id=1]     INFO    org.eclipse.jetty.server.Server#doStart: jetty-9.4.43.v20210629; built: 2021-06-30T11:07:22.254Z; git: 526006e
cfa3af7f1a27ef3a288e2bef7ea9dd7e8; jvm 11.0.7+10-LTS
2021-12-16 09:15:19.974+0000 [id=1]     INFO    o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.Jet
tyJspServlet
2021-12-16 09:15:20.102+0000 [id=1]     INFO    o.e.j.s.s.DefaultSessionIdManager#doStart: DefaultSessionIdManager workerName=node0
2021-12-16 09:15:20.102+0000 [id=1]     INFO    o.e.j.s.s.DefaultSessionIdManager#doStart: No SessionScavenger set, using defaults
2021-12-16 09:15:20.104+0000 [id=1]     INFO    o.e.j.server.session.HouseKeeper#startScavenging: node0 Scavenging every 600000ms
2021-12-16 09:15:21.314+0000 [id=1]     INFO    hudson.WebAppMain#contextInitialized: Jenkins home directory: /jenkins/jenkins found at: SystemProperties.getP
roperty("JENKINS_HOME")
2021-12-16 09:15:21.756+0000 [id=1]     INFO    o.e.j.s.handler.ContextHandler#doStart: Started w.@11ce2e22{Jenkins v2.325,/,file:///var/cache/jenkins/war/,AV
AILABLE}{/var/cache/jenkins/war}
2021-12-16 09:15:21.796+0000 [id=1]     INFO    o.e.j.server.AbstractConnector#doStart: Started ServerConnector@6771beb3{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2021-12-16 09:15:21.796+0000 [id=1]     INFO    org.eclipse.jetty.server.Server#doStart: Started @3783ms
2021-12-16 09:15:21.824+0000 [id=23]    INFO    winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled
2021-12-16 09:15:22.680+0000 [id=30]    INFO    jenkins.InitReactorRunner$1#onAttained: Started initialization
2021-12-16 09:15:25.072+0000 [id=29]    INFO    jenkins.InitReactorRunner$1#onAttained: Listed all plugins
2021-12-16 09:15:32.517+0000 [id=29]    INFO    jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
2021-12-16 09:15:32.598+0000 [id=30]    INFO    jenkins.InitReactorRunner$1#onAttained: Started all plugins
2021-12-16 09:15:35.309+0000 [id=29]    INFO    jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
2021-12-16 09:15:35.679+0000 [id=30]    INFO    hudson.slaves.SlaveComputer#tryReconnect: Attempting to reconnect Dev 2019
2021-12-16 09:15:35.692+0000 [id=30]    INFO    hudson.slaves.SlaveComputer#tryReconnect: Attempting to reconnect Project Pack 2020
2021-12-16 09:15:35.734+0000 [id=30]    INFO    hudson.slaves.SlaveComputer#tryReconnect: Attempting to reconnect Solaris11
2021-12-16 09:15:35.735+0000 [id=30]    INFO    hudson.slaves.SlaveComputer#tryReconnect: Attempting to reconnect SuSE11
2021-12-16 09:15:35.797+0000 [id=30]    INFO    jenkins.InitReactorRunner$1#onAttained: System config loaded
2021-12-16 09:15:36.394+0000 [id=18]    INFO    o.j.p.s.StrictCrumbIssuer#validateCrumb: Invalid crumb found in the request
2021-12-16 09:15:36.418+0000 [id=11]    INFO    o.j.p.s.StrictCrumbIssuer#validateCrumb: Invalid crumb found in the request
2021-12-16 09:15:36.897+0000 [id=30]    INFO    jenkins.InitReactorRunner$1#onAttained: System config adapted
2021-12-16 09:15:37.591+0000 [id=60]    WARNING h.plugins.sshslaves.SSHLauncher#checkJavaIsInPath: Java is not in the PATH nor configured with the javaPath se
tting, Jenkins will try to guess where is Java, this guess will be removed in the future. :Launch agents via SSH
2021-12-16 09:15:38.115+0000 [id=29]    INFO    jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
2021-12-16 09:15:38.132+0000 [id=30]    INFO    jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
2021-12-16 09:15:38.323+0000 [id=111]   INFO    hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Download metadata
2021-12-16 09:15:38.347+0000 [id=111]   INFO    hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Download metadata. 24 ms
2021-12-16 09:15:38.774+0000 [id=30]    INFO    jenkins.InitReactorRunner$1#onAttained: Completed initialization
2021-12-16 09:15:39.234+0000 [id=22]    WARNING j.branch.WorkspaceLocatorImpl#getWorkspaceRoot: JENKINS-2111 path sanitization ineffective when using legacy W
orkspace Root Directory ‘${ITEM_ROOTDIR}/workspace’; switch to ‘${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}’ as in JENKINS-8446 / JENKINS-21942
2021-12-16 09:15:39.281+0000 [id=22]    INFO    hudson.WebAppMain$3#run: Jenkins is fully up and running
2021-12-16 09:15:41.262+0000 [id=153]   INFO    h.TcpSlaveAgentListener$ConnectionHandler#run: Connection #1 failed: java.io.EOFException
2021-12-16 09:15:41.288+0000 [id=154]   INFO    h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #2 from /10.23.135.73:55677
2021-12-16 09:15:41.310+0000 [id=155]   INFO    h.TcpSlaveAgentListener$ConnectionHandler#run: Connection #3 failed: java.io.EOFException
2021-12-16 09:15:41.389+0000 [id=159]   INFO    h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #4 from /10.23.135.16:62618
2021-12-16 09:15:44.542+0000 [id=59]    INFO    o.j.p.m.MailWatcherNotification#log: mail-watcher-plugin: notified: Computer Dev 2019 marked online
[12/16/21 09:15:44] SSH Launch of Dev 2019 on server1r1 completed in 8,453 ms
2021-12-16 09:15:44.592+0000 [id=58]    INFO    o.j.p.m.MailWatcherNotification#log: mail-watcher-plugin: notified: Computer SuSE11 marked online
2021-12-16 09:15:44.592+0000 [id=60]    INFO    o.j.p.m.MailWatcherNotification#log: mail-watcher-plugin: notified: Computer Project Pack 2020 marked online
[12/16/21 09:15:44] SSH Launch of SuSE11 on server1sh completed in 8,558 ms
[12/16/21 09:15:44] SSH Launch of Project Pack 2020 on server2r1 completed in 8,559 ms
2021-12-16 09:15:45.213+0000 [id=200]   INFO    h.TcpSlaveAgentListener$ConnectionHandler#run: Connection #5 failed: java.io.EOFException
2021-12-16 09:15:45.268+0000 [id=201]   INFO    h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #6 from /10.23.135.63:55200
2021-12-16 09:15:45.593+0000 [id=145]   INFO    o.j.p.m.MailWatcherNotification#log: mail-watcher-plugin: notified: Computer Win2019 marked online
2021-12-16 09:15:46.366+0000 [id=151]   INFO    o.j.p.m.MailWatcherNotification#log: mail-watcher-plugin: notified: Computer Win2016 marked online
2021-12-16 09:15:48.210+0000 [id=61]    INFO    o.j.p.m.MailWatcherNotification#log: mail-watcher-plugin: notified: Computer Solaris11 marked online
[12/16/21 09:15:48] SSH Launch of Solaris11 on server4r1 completed in 12,106 ms
2021-12-16 09:15:49.183+0000 [id=52]    INFO    o.j.p.m.MailWatcherNotification#log: mail-watcher-plugin: notified: Computer RedHat CI marked online
2021-12-16 09:37:24.136+0000 [id=341]   INFO    hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Periodic background build discarder
3

That looks like a startup log not a shutdown log (due to the fact it keeps saying startup).

My recommendation would be to increase the timeout - centos - How to change systemd service timeout value? - Unix & Linux Stack Exchange

My guess would be that mail watcher plugin, its over 5 years old, and I could easily see it having some hook somewhere that isn’t letting things shutdown. I don’t know though.

4

Thank you for the suggestion. I think I gave it long enough …

$ sudo systemctl show jenkins.service -p TimeoutStopUSec
TimeoutStopUSec=35min
$ date; time sudo systemctl start jenkins
Fri 17 Dec 09:08:25 GMT 2021
Job for jenkins.service failed because a timeout was exceeded. See "systemctl status jenkins.service" and "journalctl -xe" for details.

real    35m0.089s
user    0m0.015s
sys     0m0.018s
5

Hi Tim Scott, Did you figure out what was causing this issue?

FWIW, I’m seeing the same thing… the Jenkins process is created and runs OK but the systemctl start command times out.

6

Briefly: no. Still suffering :-(.

I haven’t given priority to the investigation, though. I’ll take another look tomorrow.

7

A reminder about threadumps came up the other day on chat. It might be a decent way to debug this. If its hanging, try to get a threaddump and see whats still running.

8

May also want to try Jenkins 2.335, since it switched the Linux installers (deb, Red Hat rpm, SUSE rpm) from System V init to systemd.

9

Just updated to 2.335 - no change :frowning: .

10

Thanks, @halkeye, but I think this outside my skill set to interpret. I have the threadDump page but no idea what I’m looking for in it.
Jenkins itself is not hanging - everything seems to work fine apart from the system not recognising that it has started up (Now v2.335).
It’s as if a process is running in the foreground instead of hiving itself away in the background or returning “OK” or …

11

That may indicate that the command used to start and stop the Jenkins process on your Linux operating system is not behaving the same as other Linux operating systems.

When I run sudo systemctl status jenkins on Red Hat Enterprise Linux 8 with Jenkins 2.335 installed, it reports:

[mwaite@rhel-8-a ~]$ sudo systemctl status jenkins
● jenkins.service - Jenkins Continuous Integration Server
   Loaded: loaded (/usr/lib/systemd/system/jenkins.service; disabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/jenkins.service.d
           └─override.conf
   Active: active (running) since Thu 2022-02-17 08:07:52 MST; 49min ago
 Main PID: 1919249 (java)
    Tasks: 228 (limit: 203606)
   Memory: 5.7G
   CGroup: /system.slice/jenkins.service
           └─1919249 /opt/jdk-11/bin/java -Djava.awt.headless=true -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+UseStringDeduplicat>

Feb 17 08:50:54 rhel-8-a.markwaite.net jenkins[1919249]: 2022-02-17 15:50:54.254+0000 [id=4810]        INFO        o.j.p.g.A>
Feb 17 08:51:01 rhel-8-a.markwaite.net jenkins[1919249]: 2022-02-17 15:51:01.759+0000 [id=57]        WARNING        jenkins.>
Feb 17 08:52:01 rhel-8-a.markwaite.net jenkins[1919249]: 2022-02-17 15:52:01.747+0000 [id=57]        WARNING        jenkins.>
Feb 17 08:53:01 rhel-8-a.markwaite.net jenkins[1919249]: 2022-02-17 15:53:01.732+0000 [id=57]        WARNING        jenkins.>
Feb 17 08:53:04 rhel-8-a.markwaite.net jenkins[1919249]: 2022-02-17 15:53:04.306+0000 [id=5436]        INFO        hudson.mo>
Feb 17 08:53:07 rhel-8-a.markwaite.net jenkins[1919249]: 2022-02-17 15:53:07.279+0000 [id=5436]        INFO        hudson.mo>
Feb 17 08:54:01 rhel-8-a.markwaite.net jenkins[1919249]: 2022-02-17 15:54:01.758+0000 [id=57]        WARNING        jenkins.>
Feb 17 08:55:01 rhel-8-a.markwaite.net jenkins[1919249]: 2022-02-17 15:55:01.760+0000 [id=57]        WARNING        jenkins.>
Feb 17 08:56:01 rhel-8-a.markwaite.net jenkins[1919249]: 2022-02-17 15:56:01.740+0000 [id=57]        WARNING        jenkins.>
Feb 17 08:57:01 rhel-8-a.markwaite.net jenkins[1919249]: 2022-02-17 15:57:01.764+0000 [id=57]        WARNING        jenkins.>

What Linux variant and version are you running? Is it updated with the most recent fixes from the operating system provider? What is your output from systemctl status jenkins?

12

I’m using RH7. I believe it was patched with security updates etc in January.

My sudo systemctl status jenkins looks a little different:

$ sudo systemctl status jenkins
● jenkins.service - LSB: Jenkins Continuous Integration Server
   Loaded: loaded (/etc/rc.d/init.d/jenkins; static; vendor preset: disabled)
   Active: failed (Result: timeout) since Thu 2022-02-17 13:53:00 GMT; 2h 28min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 30264 ExecStart=/etc/rc.d/init.d/jenkins start (code=killed, signal=TERM)
   CGroup: /system.slice/jenkins.service
           ├─30269 runuser -s /bin/bash jenkins -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/jenkins...
           ├─30270 bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/jenkins/jenkins -jar /usr/lib/j...
           └─30271 /etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/jenkins/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jen...

Feb 17 13:48:00 jenkins-systemname runuser[30269]: pam_unix(runuser:session): session opened for user jenkins by (uid=0)
Feb 17 13:48:21 jenkins-systemname jenkins[30264]: Starting Jenkins WARNING: An illegal reflective access operation has occurred
Feb 17 13:48:21 jenkins-systemname jenkins[30264]: WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/var/cache/j...nalize()
Feb 17 13:48:21 jenkins-systemname jenkins[30264]: WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass
Feb 17 13:48:21 jenkins-systemname jenkins[30264]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
Feb 17 13:48:21 jenkins-systemname jenkins[30264]: WARNING: All illegal access operations will be denied in a future release
Feb 17 13:53:00 jenkins-systemname systemd[1]: jenkins.service start operation timed out. Terminating.
Feb 17 13:53:00 jenkins-systemname systemd[1]: Failed to start LSB: Jenkins Continuous Integration Server.
Feb 17 13:53:00 jenkins-systemname systemd[1]: Unit jenkins.service entered failed state.
Feb 17 13:53:00 jenkins-systemname systemd[1]: jenkins.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

Are we running into a Java version problem for a plugin? I recently had trouble with some jUnit tests with JDK 11 and I see a very similar error, here:
WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass

I’m using Corretto-11.0.7.10.1.

13

That output seems to indicate that you’re not running Jenkins 2.335. If you are running 2.335, then it somehow failed to upgrade from System V init to systemd. The line that says

should instead say something like:

I suspect that you’re seeing the same issue that was reported for Amazon Linux 2 as

https://issues.jenkins.io/browse/JENKINS-67487

Amazon Corretto 11.0.7 has been superseded by 11.0.14 (and the intervening releases), but I doubt it will make any difference in this case

1 Like
14

This might be the key - is there something other than “launch with an updated .war file” that I need to do in order to carry out the upgrade?
Have I been missing 42 steps all this time?

15

Yes, you need to use the operating system package manager (yum) to perform the update. If you change the war file that the operating system package manager installed, you’ve skipped all the processing that is performed by the operating system package manager.

16

Hey Tim!
Hope its not too late.
Recently I installed jenkins on my Ubuntu computer and faced the same problem. I googled for the error were showing while running ther jenkins service but did not found any way to fix it. after too many tries I changed the port for jenkins and it worked for me.

sudo systemctl edit jenkins

[Service]
Environment="JENKINS_PORT=8989"

sudo systemctl restart jenkins
1 Like
17

Took a little fiddling - but I think I got there.
It’s started without timing out and just behaved through this latest upgrade to 2.335 - 2.338.

I can easily see how using yum makes a huge difference over updating the .war. I guess I’m just used to updating my application by updating the .war … and was probably misled by the upgrade / download links that I didn’t think about enough.

18

Job for jenkins.service failed because a timeout was exceeded。楼主好，我使用centos7安装最新版的jenkins完成后也保存启动超时。使用命令查看也没有看到其它报错。请问楼主你的问题解决了吗？
以下为我这边的信息。解决的话麻烦帮忙看看谢谢
[root@www ~]# systemctl status jenkins.service -l
● jenkins.service - Jenkins Continuous Integration Server
Loaded: loaded (/usr/lib/systemd/system/jenkins.service; enabled; vendor preset: disabled)
Active: inactive (dead) since 六 2022-04-30 16:21:19 CST; 5min ago
Process: 69194 ExecStart=/usr/bin/jenkins (code=exited, status=143)
Main PID: 69194 (code=exited, status=143)
Status: “Jenkins stopped”

4月 30 16:20:35 主机名 jenkins[69194]: Jenkins initial setup is required. An admin user has been created and a password generated.
4月 30 16:20:35 主机名 jenkins[69194]: Please use the following password to proceed to installation:
4月 30 16:20:35 主机名 jenkins[69194]: bdbb66675db44738a057b28a84344564
4月 30 16:20:35 主机名 jenkins[69194]: This may also be found at: /var/lib/jenkins/secrets/initialAdminPassword
4月 30 16:21:19 主机名 systemd[1]: Stopped Jenkins Continuous Integration Server.

19

Assuming you’re running a recent Jenkins weekly release or a recent Jenkins LTS release, then you may have made a mistake when upgrading. Upgrades of packages installed on CentOS should be performed with

# yum upgrade

When I run systemctl status jenkins with Jenkins on Red Hat Enterprise Linux 8, I see:

● jenkins.service - Jenkins Continuous Integration Server
   Loaded: loaded (/etc/systemd/system/jenkins.service; disabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/jenkins.service.d
           └─override.conf
   Active: active (running) since Fri 2022-04-29 15:12:49 MDT; 16h ago

If a yum upgrade does not resolve your issue, please provide more details about your configuration so that others can assist.

20

Try another version, reinstall an older version
I had the same problem as you, and it worked fine after reinstalling version 2.303.3

21

Thanks @OldMan-3301
Out of all suggestions/solutions, your answer solved my jenkins problem.