My Jenkins startup times out. It appears to work, but the start command times out on the shell prompt

Using Jenkins
22

Hi @MarkEWaite, I’m facing a similar issue as @gulbrain but don’t easily have the option to yum upgrade, mainly because the available version via yum for me is still 2.235.3-1.1 - which is the version I sought to upgrade from.

I’m running on RHEL 7.9 Maipo.

I’d read it was okay to use either yum or manually install the war. I guess this is no longer the case :frowning:

I went from 2.235.3-1.1 (java 8/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64) to 2.361.1 (java 11/Red_Hat-11.0.16.0.8-1.el7_9) and had some issues with plugins (matrix security and LDAP) that I eventually seem to have resolved. In the process I downgraded to 2.263.1, then back to the original 2.235.3-1.1, with no luck, and finally got 2.361.1 working after trying a bunch of things like deleting authentication strategies from config.xml and allowing anonymous access, to be able to access the UI and fix the plugin issues.

Currently, it’s LDAP protected, 2.361.1, plugins are all up to date (and so far at least one of my jobs is working correctly, I haven’t extensively tested), but I face the same issue, where systemctl start jenkins times out, and systemctl stop jenkins fails to do anything.

The java process looks different too, before I believe it ran as a single process as the jenkins user, now there are 3 java processes, one running as root, and 2 as jenkins.

There seem to be some other issues, systemctl status jenkins mentions a problem with org.codehaus.groovy.vmplugin.v7.Java7$1

I don’t have the option to switch the port as there is a loadbalancer in front expecting to connect on 8080.

Is there an idea of what might be going on? I’m wondering what the yum install might be doing that the manual war install doesn’t do, or breaks? Maybe something to do with the jenkins user?

My install seems to be using system.d, I believe the timeout is 5 mins, which is when the systemctl start jenkins command prompt returns with a failure notice, though the java process(es) continue to run, and jenkins appears to be working, though systemctl stop jenkins will not stop it (pkill java does)

Thanks for any additional info!


[root@localhost:~]$ systemctl status jenkins
● jenkins.service - LSB: Jenkins Automation Server
   Loaded: loaded (/etc/rc.d/init.d/jenkins; bad; vendor preset: disabled)
  Drop-In: /etc/systemd/system/jenkins.service.d
           └─override.conf
   Active: failed (Result: timeout) since Thu 2022-09-08 21:54:26 EDT; 34min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 10975 ExecStart=/etc/rc.d/init.d/jenkins start (code=killed, signal=TERM)
   CGroup: /system.slice/jenkins.service
           ├─10980 runuser -s /bin/bash jenkins -c ulimit -S -c 0 >/dev/null 2>&1 ; /usr/lib/jvm/jre-11/bin/java -Djava.awt.headless=true -Djava.io.tmpdir=/var/lib/jenkins/tmp -DJENKINS_HOME=/var/lib/jenkin...
           ├─10981 bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /usr/lib/jvm/jre-11/bin/java -Djava.awt.headless=true -Djava.io.tmpdir=/var/lib/jenkins/tmp -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/...
           └─10982 /usr/lib/jvm/jre-11/bin/java -Djava.awt.headless=true -Djava.io.tmpdir=/var/lib/jenkins/tmp -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/je...
Sep 08 21:49:26 localhost runuser[10980]: pam_unix(runuser:session): session opened for user jenkins by (uid=0)
Sep 08 21:49:33 localhost jenkins[10975]: Starting Jenkins WARNING: An illegal reflective access operation has occurred
Sep 08 21:49:33 localhost jenkins[10975]: WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/var/cache/jenkins/war/WEB-INF/lib/groovy-all-2....g.Class,int)
Sep 08 21:49:33 localhost jenkins[10975]: WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1
Sep 08 21:49:33 localhost jenkins[10975]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
Sep 08 21:49:33 localhost jenkins[10975]: WARNING: All illegal access operations will be denied in a future release
Sep 08 21:54:26 localhost systemd[1]: jenkins.service start operation timed out. Terminating.
Sep 08 21:54:26 localhost systemd[1]: Failed to start LSB: Jenkins Automation Server.
Sep 08 21:54:26 localhost systemd[1]: Unit jenkins.service entered failed state.
Sep 08 21:54:26 localhost systemd[1]: jenkins.service failed.
Hint: Some lines were ellipsized, use -l to show in full.


root     10980     1  0 21:49 ?        00:00:00 runuser -s /bin/bash jenkins -c ulimit -S -c 0 >/dev/null 2>&1 ; /usr/lib/jvm/jre-11/bin/java -Djava.awt.headless=true -Djava.io.tmpdir=/var/lib/jenkins/tmp -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=8080 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20
jenkins  10981 10980  0 21:49 ?        00:00:00 bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /usr/lib/jvm/jre-11/bin/java -Djava.awt.headless=true -Djava.io.tmpdir=/var/lib/jenkins/tmp -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=8080 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20
jenkins  10982 10981  3 21:49 ?        00:01:43 /usr/lib/jvm/jre-11/bin/java -Djava.awt.headless=true -Djava.io.tmpdir=/var/lib/jenkins/tmp -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=8080 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20
23

Your configuration seems wrong. When I installed Jenkins 2.361.1 on a CentOS 7 machine based on the CentOS 7 instructions

When I run with that installation, the output of systemctl status jenkins

● jenkins.service - Jenkins Continuous Integration Server
   Loaded: loaded (/usr/lib/systemd/system/jenkins.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2022-09-09 19:05:43 UTC; 1min 1s ago
 Main PID: 27848 (java)
   CGroup: /system.slice/jenkins.service
           └─27848 /usr/bin/java -Djava.awt.headless=true -jar /usr/share/java/jenkins.war --webroot=%C/jenkins/war --httpPort=8080

Sep 09 19:05:28 gcp-centos-7-a-las-vegas jenkins[27848]: *************************************************************
Sep 09 19:05:28 gcp-centos-7-a-las-vegas jenkins[27848]: *************************************************************
Sep 09 19:05:43 gcp-centos-7-a-las-vegas jenkins[27848]: 2022-09-09 19:05:43.939+0000 [id=35]        INFO        jenkins.InitReactorRunner$1#onAttained: Completed initialization
Sep 09 19:05:43 gcp-centos-7-a-las-vegas jenkins[27848]: 2022-09-09 19:05:43.963+0000 [id=23]        INFO        hudson.lifecycle.Lifecycle#onReady: Jenkins is fully up and running
Sep 09 19:05:43 gcp-centos-7-a-las-vegas systemd[1]: Started Jenkins Continuous Integration Server.
Sep 09 19:05:44 gcp-centos-7-a-las-vegas jenkins[27848]: 2022-09-09 19:05:44.313+0000 [id=51]        INFO        h.m.DownloadService$Downloadable#load: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller
Sep 09 19:05:44 gcp-centos-7-a-las-vegas jenkins[27848]: 2022-09-09 19:05:44.314+0000 [id=51]        INFO        hudson.util.Retrier#start: Performed the action check updates server successfully at the attempt #1
Sep 09 19:05:44 gcp-centos-7-a-las-vegas jenkins[27848]: 2022-09-09 19:05:44.317+0000 [id=51]        INFO        hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Download metadata. 16,904 ms
Sep 09 19:05:47 gcp-centos-7-a-las-vegas jenkins[27848]: 2022-09-09 19:05:47.708+0000 [id=59]        INFO        hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Periodic background build discarder
Sep 09 19:05:47 gcp-centos-7-a-las-vegas jenkins[27848]: 2022-09-09 19:05:47.710+0000 [id=59]        INFO        hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Periodic background build discarder. 0 ms

Notice that there is no mention of /etc/rc.d/init.d/jenkins in that output.

24

this makes me think its still some initd scripts, and not the new systemd rpms

25

I suspect that is the most important issue to resolve. If installed with yum, upgrade with yum.

26

Thanks @halkeye and @MarkEWaite. I wonder if there is a way to do what the yum/OS version of the install does, manually, or a way to figure that out?

I found that my jenkins repo was disabled, so I could probably enable it and upgrade jenkins to 2.361.1 with yum. However, I’m wondering if all of my previous steps might prevent this from working?

For instance, I went from 2.235.3-1.1 (no systemd?) to war file upgrade 2.361.1 and then went down to war install 2.263.1 and then back up up to war install 2.361.1 - all manual war installs - so I guess no systemv to systemd conversion was attempted? - the initial 2.235.3-1.1 install was via yum, I’m pretty sure.

Thanks again for your help.

27

I’m willing to bet its not tested or supported.

I’m pretty sure you can either use java -jar jenkins.jar manually, or use the scripts. If you want to use the maintained and tested scripts, you’ll want to use rpm and upgrade via yum (or download rpm directly), and not update war file.

Note, I use docker so I don’t know the packaging that welll

28

Thanks for the info.

I shutdown my working 2.361.1 instance (working that is, except for the systemd/systemctl stop script, weird systemctl status jenkins message, and weird number of java processes running), with pkill java, kept my /etc/sysconfig/jenkins file configured to use java 11, put back the old 2.235.3 /usr/lib/jenkins.war file, as well as the old 2.235.3 /opt/jenkins/config.xml file, removed the somehow created /etc/systemd/system/jenkins.service.d/override.conf file, which only had the following:

[Service]
Environment="JAVA_OPTS=-Djava.awt.headless=true -Djava.io.tmpdir=/var/lib/jenkins/tmp"

and ran:

sudo yum --enablerepo=jenkins update jenkins

This enabled my disabled jenkins repo temporarily, and updated Jenkins successfully from 2.235.3 to 2.361.1.

The only change I could determine (between the manual install and the yum update jenkins install), so far is that /etc/systemd/system/jenkins.service.d/override.conf now has:


[Service]
Environment="JENKINS_JAVA_CMD=/usr/lib/jvm/jre-11/bin/java"
Environment="JAVA_OPTS=-Djava.awt.headless=true -Djava.io.tmpdir=/var/lib/jenkins/tmp"

JENKINS_JAVA_CMD is the param value I’d placed in my /etc/sysconfig/jenkins file to update java 8 to 11. I’d done this during the manual install failure. Could this have been the only thing stopping the manual update from working?

I found this article helpful Linux installation packages migrated from System V init to systemd

Thanks again for your help!

29

Possibly, though there are multiple things that happen in the upgrade. When you replaced the jenkins.war file, you skipped all the steps that the upgrade script would perform. I’m not ready to make any guesses why bypassing the upgrade script would not work.

30

/lib/systemd/system/jenkins.service modify TimeoutStartSec=infinity

31

Please don’t edit that file. On Ubuntu 22.04, that file includes a heading that says:

This file is managed by systemd(1). Do NOT edit this file manually!
To override these settings, run:

systemctl edit jenkins

If you need to adjust the value of TimeoutStartSec from its default of 90, please use systemctl edit jenkins to do it.

More information is available at:

32

I had been doing that for a while. Where I think I fell foul was that this did not update the system startup/shutdown scripts - thus, the problem. You may get away with updating the war file for some time, but one day the system script requirements might change and then you’ll have to re-write your upgrade process. It probably doesn’t help that when the upgrade is available, the link directs users to the war file download.

I have arrived at the following (after my IT people removed the Jenkins repo from the system):

# Retrieve the weekly update repo:
sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat/jenkins.repo

# Import the key:
sudo rpm --import https://pkg.jenkins.io/redhat/jenkins.io.key

# Look for an update
sudo yum --enablerepo=jenkins --downloadonly update jenkins

# While ^^ runs, go to Manage Jenkins and prepare for shutdown.

# If the above finds an update:
sudo service jenkins stop
sudo yum --assumeyes localupdate /var/cache/yum/x86_64/7Server/jenkins/packages/*.rpm
sudo service jenkins start

# If no update is available, cancel the shutdown prep.

This has been working OK for me since at least October 2022 when I last amended the update script.

Of course, if you use a lot of Jenkins features and plugins you will want to test the upgrade before making a commitment ion your real build system. I’ll leave that workflow to the reader to consider rollback processes and/or test servers: or not to upgrade.