LTS Jenkins upgrade from 2.263.4 to 2.319.3

Using Jenkins Ask a question
1

I’m running jenkins as .war file on RHEL 7.4 / Java 8
It’s being launched as service (init file). I replaced current jenkins.war with newer LTS version (2.319.3) and after starting service command prompt got frozen. Ctrl+C & I checked that UI is available. But after checking running processes I discovered that instead of 1 I’ve got 3 and one running as root.

jenkins@myhost-25v:/usr/lib/jenkins$ ps aux | grep java
root     155755  0.0  0.0 148684  1720 ?        S    12:22   0:00 runuser -s /bin/bash jenkins -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/java -Djava.awt.headless=true -Djava.io.tmpdir=/jenkins/tmp -Xmx4g -Xms2g -XX:+UseConcMarkSweepGC "-Dhudson.model.DirectoryBrowserSupport.CSP=script-src 'unsafe-inline';" -DJENKINS_HOME=/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=8080 --httpListenAddress=0.0.0.0 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20
jenkins  155756  0.0  0.0 113128  1412 ?        Ss   12:22   0:00 bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/java -Djava.awt.headless=true -Djava.io.tmpdir=/jenkins/tmp -Xmx4g -Xms2g -XX:+UseConcMarkSweepGC "-Dhudson.model.DirectoryBrowserSupport.CSP=script-src 'unsafe-inline';" -DJENKINS_HOME=/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=8080 --httpListenAddress=0.0.0.0 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20
jenkins  155757  197 10.6 8075316 847804 ?      Sl   12:22   1:10 /etc/alternatives/java -Djava.awt.headless=true -Djava.io.tmpdir=/jenkins/tmp -Xmx4g -Xms2g -XX:+UseConcMarkSweepGC -Dhudson.model.DirectoryBrowserSupport.CSP=script-src 'unsafe-inline'; -DJENKINS_HOME=/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=8080 --httpListenAddress=0.0.0.0 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20
jenkins  155937  0.0  0.0 112660   944 pts/0    S+   12:23   0:00 grep java

At the moment I was not able to manage application service other than as root.
Once all processes were killed I downgraded to previously running 2.263.4 and started - all is fine. Then I used another LTS: 2.303.3 and stated - all is working fine.

I’ve seen that latest LTS Jenkins installer on linux switched to systemd but is that possible that “war” version can be affected by that?
Does anyone experienced something similar or have any idea why I’m not able to launch anything higher than 2.303.3?

2

Those look a lot like scripts an os run.

When you say war version. Can you walk us through how you are upgrading and how you are starting up Jenkins?

I havnt spent any time looking at the packaging scripts (I use docker myself) but the ps output you posted looks like something is calling something that is running Jenkins as the Jenkins user.

I feel like those scripts need to be updated to use exec but that’s a different issue (unless you wrote them)

3

Procedure is simple and all is being done as Jenkins user:

stop service > rename jenkins.war > wget https://get.jenkins.io/war-stable/<lts-version>/jenkins.war > chmod 644 jenkins.war > start service

Same sysconfig & init script only difference is jenkins.war

Exactly that is init script, even though launched as root will launch application as application user. But it went that application user starting service is executing something as root - no idea. Especially that as soon as I’ll use previous LTS it will work w/o issues.

Init script: https://pastebin.mozilla.org/7vdSHfhG

4

If you used those steps today, then you downloaded Jenkins 2.332.1. You can check the version of the war file that you’ve downloaded with the command

$ java -jar jenkins.war --version

Please don’t use the Linux installer (rpm package in your case) and then replace the war file as an upgrade technique. That bypasses the benefits of the Linux installer because it skips the upgrade steps that the Linux installer provides as part of the Linux package.

Use the Linux package manager to upgrade. For example:

# yum update jenkins

If you want to remain with Jenkins 2.319.3 rather than use the most recent 2.332.1, then you’ll need to investigate the yum syntax that will allow you to install a specific version. The Linux installer switch to systemd does not happen until the 2.332.1 LTS release. Jenkins 2.319.3 uses System V init just as previous versions did.

I hope you’ll also consider upgrading from RHEL 7.4 to RHEL 7.9. RHEL 7.4 is a very old version and likely has many known security issues that can be resolved by updating to the most recent RHEL 7 release.

You should also assure that you’re running a recent Java 8 version (preferably 8u322).

5

Hi @MarkEWaite, thanks for good spot about Jenkins being installed by rpm first. Need to rework that! Upgrades are in place.

Knowing that it looks like installed via rpm Jenkins version 2.164.3 - as I’ve found that this one was installed in past - used some settings which are not working with version higher than: 2.303.3.

Thanks to all for help!

6

Being vague about bugs doesn’t help future you or anyone else. Its worth reporting a bug via issue tracker if something is broken.

I’ve found my own threads too many times to not chuckle at