Did I overwrite my old version of Jenkins?

Using Jenkins Ask a question
1

Hi All,

My Jenkins is running on CentOS 7 and I was looking to upgrade my Jenkins version 2.277.4 to 2.375.1. I tried using sudo yum update jenkins to update jenkins. I received this message: “Public key for jenkins-2.375.1-1.1.noarch.rpm is not installed”. I read an article telling me to pull the key from online, so I performed these commands:

sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo --no-check-certificate
sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
sudo yum update jenkins

Afterwards, I haven’t been able to correctly start Jenkins. Obviously, I should have just uploaded the new war file instead of using yum update.

Any time I try Sudo systemctl start jenkins I get this warning.
Warning: jenkins.service changed on disk. Run ‘systemctl daemon-reload’ to reload units.

Any suggestions on fixing this issue? Any way to revert back to the old version?

Thank you ahead of time!

2

you did the right thing, you absolutely should have updated the rpm instead of just flat out updating the war file,

277 to 375 is almost 100 weeks apart, so 2 years of changes. You generally should be upgrading in smaller more recent versions and reading the upgrade guide to have a more smooth expeirence.

In this case, its probably due to Jenkins switching from using init.d to systemd in its more recent versions. Probably your JENKINS_HOME was overridden in /etc/default/jenkins or somewhere else, and the systemd version doesn’t read that file at all.

Canned response follows, which should give you more context:

Jenkins 2.332.1 switched Jenkins from using System V init to use systemd with its Linux package installers for Debian, Ubuntu, Red Hat, Alma, openSUSE, Rocky, and more. The LTS Upgrade Guide describes that transition and how to adapt your environment to the transition.

There is a blog post about it as well at

There is also a video introduction for RPM based distributions like Red Hat Enterprise Linux, Alma Linux, Rocky Linux, Oracle Linux, and Amazon Linux.

There is also a video introduction for deb based distributions like Debian and Ubuntu

1 Like
3

Thank you for the information! After reading through the documentation and watching the CentOS video, I tried reloading the daemon and restarting the service. However, my start/restart is timing out even though I’ve set the timer to 10 mins.

Additionally, I installed java-11-openjdk, would that cause issues?

Please find the results below.

jenkins.service - Jenkins Continuous Integration Server
Loaded: loaded (/usr/lib/systemd/system/jenkins.service; disabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/jenkins.service.d
└─override.conf
Active: activating (start) since Tue 2023-01-03 20:56:15 EST; 32s ago
Main PID: 46037 (java)
Status: “Jenkins stopped”
Tasks: 49
Memory: 1.0G
CGroup: /system.slice/jenkins.service
└─46037 /usr/bin/java -Xmx4096m -Djava.awt.headless=true -jar /usr/share/java/jenkins.war --webroot=%C/jenkins/war --httpPort=8080

Jan 03 20:56:39 vtlinuxdevauto jenkins[46037]: at hudson.model.Hudson.(Hudson.java:82)
Jan 03 20:56:39 vtlinuxdevauto jenkins[46037]: at hudson.WebAppMain$3.run(WebAppMain.java:247)
Jan 03 20:56:39 vtlinuxdevauto jenkins[46037]: 2023-01-04 01:56:39.669+0000 [id=57] SEVERE h.i.i.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler#uncaughtException: A thread (Timer-0/57) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.
Jan 03 20:56:39 vtlinuxdevauto jenkins[46037]: java.lang.IllegalStateException: Jenkins.instance is missing. Read the documentation of Jenkins.getInstanceOrNull to see what you are doing wrong.
Jan 03 20:56:39 vtlinuxdevauto jenkins[46037]: at jenkins.model.Jenkins.get(Jenkins.java:813)
Jan 03 20:56:39 vtlinuxdevauto jenkins[46037]: at hudson.model.Descriptor.getConfigFile(Descriptor.java:938)
Jan 03 20:56:39 vtlinuxdevauto jenkins[46037]: at hudson.model.Descriptor.save(Descriptor.java:910)
Jan 03 20:56:39 vtlinuxdevauto jenkins[46037]: at org.jenkinsci.plugins.ghprb.GhprbTrigger$DescriptorImpl$1.run(GhprbTrigger.java:882)
Jan 03 20:56:39 vtlinuxdevauto jenkins[46037]: at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
Jan 03 20:56:39 vtlinuxdevauto jenkins[46037]: at java.base/java.util.TimerThread.run(Timer.java:506)

4

Please find additional information below:
jenkins.service - Jenkins Continuous Integration Server
Loaded: loaded (/usr/lib/systemd/system/jenkins.service; disabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/jenkins.service.d
└─override.conf
Active: failed (Result: start-limit) since Tue 2023-01-03 21:07:43 EST; 3s ago
Process: 3006 ExecStart=/usr/bin/jenkins (code=killed, signal=TERM)
Main PID: 3006 (code=killed, signal=TERM)

Jan 03 21:07:42 vtlinuxdevauto systemd[1]: Failed to start Jenkins Continuous Integration Server.
Jan 03 21:07:42 vtlinuxdevauto systemd[1]: Unit jenkins.service entered failed state.
Jan 03 21:07:42 vtlinuxdevauto systemd[1]: jenkins.service failed.
Jan 03 21:07:43 vtlinuxdevauto systemd[1]: jenkins.service holdoff time over, scheduling restart.
Jan 03 21:07:43 vtlinuxdevauto systemd[1]: Stopped Jenkins Continuous Integration Server.
Jan 03 21:07:43 vtlinuxdevauto systemd[1]: start request repeated too quickly for jenkins.service
Jan 03 21:07:43 vtlinuxdevauto systemd[1]: Failed to start Jenkins Continuous Integration Server.
Jan 03 21:07:43 vtlinuxdevauto systemd[1]: Unit jenkins.service entered failed state.
Jan 03 21:07:43 vtlinuxdevauto systemd[1]: jenkins.service failed.

5

its going to be higher in the error logs than this. I would recommend looking at the logs via journalctl instead of just the last 10 lines via systemctl

You may want to reach out to a vendor to get some real time support. Remotely debugging these things can be pretty time intensive, and most of us pop on when we have free time.

1 Like
6

Hi Halkeye,

Do you have any suggested vendors or ways I can revert the upgrade?

Thank you

7

No sorry. The only vendor I know of is cloudbees. I tried to collect a list a while ago but didn’t have much traction.

As for downgrade. You can probably specify the version you want to yum.

8

Be extra careful if you downgrade, since most likely a lot of your configuration files have been updated to the new version, you are likely to hit incompatibilities.
The best course of action for you right now would be to have a backup of your JENKINS_HOME taken prior to the upgrade that you can revert to.

9

Agreed on the backup, but if Jenkins hasn’t successfully run, I figured the chances of config files getting updated being Low

Another option would be to start up the new version. Get it working. Then add in a couple jobs at a time to the jobs directory. But that would be essentially starting again.

I would bet most likely it’s old plugins no longer starting up. You can use the plugin manager cli to update plugins from the CLI

1 Like
10

I would rather try and get the new Jenkins version up and running before reverting to the old version. You think issue is most likely due to the old plugins? How would I go about disabling them? Do you have documentation?

11

the jenkins log (via journalctl so you get the full log) will tell you explicitly why jenkins isn’t starting up properly

12

Hi Halkeye,

Thank you for the information. I’m looking for help from a vendor, but until then I’m looking through the logs. Here’s some of the logs I’m seeing:

java.lang.RuntimeException: unsupported Java version: 11
at org.jruby.RubyInstanceConfig.initGlobalJavaVersion(RubyInstanceConfig.java:1674)
at org.jruby.RubyInstanceConfig.(RubyInstanceConfig.java:1387)
Caused: java.lang.ExceptionInInitializerError
at org.jruby.embed.internal.AbstractLocalContextProvider.(AbstractLocalContextProvider.java:42)
at org.jruby.embed.internal.SingleThreadLocalContextProvider.(SingleThreadLocalContextProvider.java:43)
at org.jruby.embed.ScriptingContainer.getProviderInstance(ScriptingContainer.java:242)
at org.jruby.embed.ScriptingContainer.(ScriptingContainer.java:226)
at org.jruby.embed.ScriptingContainer.(ScriptingContainer.java:192)
at org.kohsuke.stapler.jelly.jruby.JRubyFacet.(JRubyFacet.java:65)
at ruby.RubyRuntimePlugin.registerJRubyFacet(RubyRuntimePlugin.java:39)
at ruby.RubyRuntimePlugin.start(RubyRuntimePlugin.java:30)
at hudson.ClassicPluginStrategy.startPlugin(ClassicPluginStrategy.java:417)
at hudson.ClassicPluginStrategy.load(ClassicPluginStrategy.java:406)
Caused: java.io.IOException: Failed to initialize
at hudson.ClassicPluginStrategy.load(ClassicPluginStrategy.java:409)
at hudson.PluginManager$2$1$1.run(PluginManager.java:549)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:177)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)
at jenkins.model.Jenkins$5.runTask(Jenkins.java:1161)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:70)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-01-04 02:04:55.195+0000 [id=38] WARNING n.b.j.internal.common.JavaLogger#warn: exception while reading counters data from files in /var/lib/jenkins/monitoring/_vtlinuxdevau
java.io.EOFException: Unexpected end of ZLIB input stream
at java.base/java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:245)
at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:159)
at java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:118)
at java.base/java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2893)
at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2909)
at java.base/java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3684)
at java.base/java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:3476)
at java.base/java.io.ObjectInputStream.readString(ObjectInputStream.java:2064)

13

Again why I’m strongly recommending for the future more often upgrades, of smaller magnitude so you can read the upgrade guide.

1 Like
14

Hi All,

Thank you for the help. My team and I took a snapshot of the change and reverted back to the old version by undoing the changes.

For anyone using CentOS, you can revert by installing screen: “yum install screen”. Then checking history: “yum history”. Then undoing updates: yum history undo <#>"

I was able to revert back without any issues.