Jenkins as a service restarts by itself after upgrade from 2.303.2 to 2.319.3

Hi,

I upgraded Java from OpenJDK 8 to OpenJDK 11 on my Jenkins machine, which was the prerequisite for Jenkins upgrade.

Then I upgraded Jenkins (running as a service) on this server from version 2.235.5 to 2.303.2 (by just replacing .war file and restarting the service), which seemed to work pretty stable.
I tried to upgrade even to 2.319.3 or to 2.332.3 and I was able to log in to GUI with my AD credentials after upgrade, however there are some issues which cause Jenkins to restart by itself from time to time when using these 2 releases, so for now I downgraded it back to 2.303.2.

The question is why is this happening ?

When there was Jenkins 2.319.3 or 2.332.3 running, I could have observed the following messages in Jenkins log:

[…]
2022-06-01 11:10:26.854+0000 [id=30] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization
2022-06-01 11:10:27.484+0000 [id=33] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins
2022-06-01 11:10:42.114+0000 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
2022-06-01 11:10:42.160+0000 [id=36] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins
2022-06-01 11:10:42.227+0000 [id=35] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
2022-06-01 11:10:46.047+0000 [id=30] INFO jenkins.InitReactorRunner$1#onAttained: System config loaded
2022-06-01 11:10:47.060+0000 [id=33] INFO jenkins.InitReactorRunner$1#onAttained: System config adapted
2022-06-01 11:10:47.060+0000 [id=33] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
2022-06-01 11:10:47.066+0000 [id=36] INFO jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
2022-06-01 11:10:47.110+0000 [id=57] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Download metadata
2022-06-01 11:10:47.168+0000 [id=57] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Download metadata. 49 ms
2022-06-01 11:10:47.398+0000 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization
2022-06-01 11:10:47.524+0000 [id=23] INFO hudson.WebAppMain$3#run: Jenkins is fully up and running
2022-06-01 11:11:48.936+0000 [id=16] WARNING o.j.p.workflow.job.WorkflowRun#checkouts: JENKINS-26761: no checkouts in startTestEsign #43
2022-06-01 11:11:48.997+0000 [id=16] WARNING o.j.p.workflow.job.WorkflowRun#checkouts: JENKINS-26761: no checkouts in startTestEsign #24
2022-06-01 11:11:53.601+0000 [id=25] INFO winstone.Logger#logInternal: JVM is terminating. Shutting down Jetty
Running from: /storage/jenkins/jenkins.war
webroot: System.getProperty(“JENKINS_HOME”)
2022-06-01 11:11:55.083+0000 [id=1] INFO org.eclipse.jetty.util.log.Log#initialized: Logging initialized @977ms to org.eclipse.jetty.util.log.JavaUtilLog
2022-06-01 11:11:55.327+0000 [id=1] INFO winstone.Logger#logInternal: Beginning extraction from war file
[…]

This doesn’t happen however, when I simply start Jenkins with:
java -jar jenkins.war

What may be the issue ?
Please help with it.

Thanks in advance!
Krzysztof

Did you intially install this with downloading a .war? if you used the package management you should be updating with package upgrades. There was a fairly major change recently to switch to systemd, and i guess that the old init scripts don’t detect that things started up properly and shut things down.

1 Like

What is your distro @nowackr1 ?

OS is RH 7.9, I don’t know how exactly this Jenkins was installed - it was done by someone else but ‘rpm -qa | grep -i jenkins’ command doesn’t show anything and ‘yum list | grep -i jenkins’ shows only this:
python2-jenkins.noarch 0.4.16-2.el7 1cvmware_EPEL_EPEL_7Server
python2-jenkins-job-builder.noarch
python36-jenkins.noarch 0.4.16-2.el7 1cvmware_EPEL_EPEL_7Server

1 Like

so what command does work vs what doesn’t?

2.332 upgraded from initd to systemd, but that should only affect os packaging.

I would recommend trying out Redhat Jenkins Packages and installing it via os package.

2 Likes

On the system where it’s installed, Jenkins is running as a service:

Blockquote
# systemctl status jenkins
● jenkins.service - Jenkins Daemon
Loaded: loaded (/etc/systemd/system/jenkins.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2022-06-01 16:26:44 EEST; 16h ago
Process: 108075 ExecStart=/usr/bin/java -Djava.awt.headless=true -DJENKINS_HOME=/storage/jenkins -jar /storage/jenkins/jenkins.war --daemon --logfile=/storage/jenkins/jenkins.log --httpPort=9090 (code=exited, status=0/SUCCESS)
Main PID: 108095 (java)
Tasks: 61
Memory: 3.3G
CGroup: /system.slice/jenkins.service
└─108095 /usr/bin/java -Dcom.sun.akuma.Daemon=daemonized -Djava.awt.headless=true -DJENKINS_HOME=/storage/jenkins -jar /storage/jenkins/jenkins.war --daemon --logfile=/storage/jenkins/jenkins.log -…

Blockquote
# ps -ef | grep jenkins
jenkins 108095 1 0 Jun01 ? 00:05:41 /usr/bin/java -Dcom.sun.akuma.Daemon=daemonized -Djava.awt.headless=true -DJENKINS_HOME=/storage/jenkins -jar /storage/jenkins/jenkins.war --daemon --logfile=/storage/jenkins/jenkins.log --httpPort=9090

As I wrote - this is running fine when I put jenkins.war version 2.303.2 but when I replace it with 2.319.3 or 2.332.3, then it’s restarting with “JVM is terminating. Shutting down Jetty” visible in Jenkins log. However I don’t experience such issue when I just start Jenkins using ‘java -jar jenkins.war’ command.

By the way - when I start Jenkins 2.319.3 with ‘systemctl start jenkins’ command, it looks like this:

Blockquote
# systemctl status jenkins
● jenkins.service - Jenkins Daemon
Loaded: loaded (/etc/systemd/system/jenkins.service; enabled; vendor preset: disabled)
Active: activating (start) since Thu 2022-06-02 09:44:25 EEST; 5s ago
Main PID: 108095 (code=exited, status=143); : 28612 (java)
Tasks: 45
Memory: 375.2M
CGroup: /system.slice/jenkins.service
└─28612 /usr/bin/java -Djava.awt.headless=true -DJENKINS_HOME=/storage/jenkins -jar /storage/jenkins/jenkins.war --daemon --logfile=/storage/jenkins/jenkins.log --httpPort=9090
Jun 02 09:44:25 systemd[1]: Starting Jenkins Daemon…

Blockquote
# ps -ef | grep jenkins
jenkins 28612 1 61 09:44 ? 00:00:41 /usr/bin/java -Djava.awt.headless=true -DJENKINS_HOME=/storage/jenkins -jar /storage/jenkins/jenkins.war --daemon --logfile=/storage/jenkins/jenkins.log --httpPort=9090

I noticed when Jenkins 2.319.3 starts, config.xml content is a bit different than for 2.303.2:

Blockquote
diff /storage/jenkins/config.xml.2.303.2 /storage/jenkins/config.xml.2.319.3
4c4
< <version>2.303.2</version>
-—
> <version>2.319.3</version>
186a187
> <nodeRenameMigrationNeeded>true</nodeRenameMigrationNeeded>

I also noticed the following warnings popping up with Jenkins 2.319.3:

Blockquote
# systemctl status -l jenkins
● jenkins.service - Jenkins Daemon
Loaded: loaded (/etc/systemd/system/jenkins.service; enabled; vendor preset: disabled)
Active: activating (start) since Thu 2022-06-02 10:52:15 EEST; 18s ago
Main PID: 108095 (code=exited, status=143); : 103064 (java)
Tasks: 50
Memory: 1.1G
CGroup: /system.slice/jenkins.service
└─103064 /usr/bin/java -Djava.awt.headless=true -DJENKINS_HOME=/storage/jenkins -jar /storage/jenkins/jenkins.war --daemon --logfile=/storage/jenkins/jenkins.log --httpPort=9090
Jun 02 10:52:15 systemd[1]: Starting Jenkins Daemon…
Jun 02 10:52:23 java[103064]: WARNING: An illegal reflective access operation has occurred
Jun 02 10:52:23 java[103064]: WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$2 (file:/storage/jenkins/war/WEB-INF/lib/guice-4.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte,int,int,java.security.ProtectionDomain)
Jun 02 10:52:23 java[103064]: WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$2
Jun 02 10:52:23 java[103064]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
Jun 02 10:52:23 java[103064]: WARNING: All illegal access operations will be denied in a future release