Jenkins not starting after upgrade, Required plugin is disabled errors

Hi Folks,

I attempted an upgrade from Jenkins 2.319.3 to 2.414.1, and I’m running into some errors. I have a backup and I’m going to roll back, but I’m wondering if I have been doing something wrong so that I can get it right next time.

I upgraded jenkins (latest LTS) on my linux server using yum. The package installed fine.

The server was running java 8, so I also installed java 11 (also using yum) and did the following:

systemctl edit jenkins
-Added an environment entry for JAVA_HOME and JENKINS_JAVA_CMD, both pointing to the java 11 location (at present I’d like to leave the alternatives java link pointing to java 8 as there are other dependencies on the server)

-Started jenkins

The server fails to start.

Looking at the logs, I’m seeing this on startup:

Sep 08 19:42:09 jenkins[317022]: 2023-09-08 09:42:09.633+0000 [id=29] SEVERE jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Authentication Tokens API Plugin v>
Sep 08 19:42:09 jenkins[317022]: java.io.IOException: Failed to load: Authentication Tokens API Plugin (authentication-tokens 1.4)
Sep 08 19:42:09 jenkins[317022]: - Required plugin is disabled: Credentials Plugin (credentials)

…and then a lot of similar entries, many of them saying a Required plugin is disabled.

In the end the service jenkins start command times out and the service says it isn’t starting.

I can get to the Jenkins console via a browser, but it throws java errors:

com.thoughtworks.xstream.mapper.CannotResolveClassException: hudson.security.ProjectMatrixAuthorizationStrategy
at com.thoughtworks.xstream.mapper.DefaultMapper.realClass(DefaultMapper.java:81)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
at com.thoughtworks.xstream.mapper.DynamicProxyMapper.realClass(DynamicProxyMapper.java:55)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
at com.thoughtworks.xstream.mapper.PackageAliasingMapper.realClass(PackageAliasingMapper.java:88)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
at com.thoughtworks.xstream.mapper.ClassAliasingMapper.realClass(ClassAliasingMapper.java:79)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
at com.thoughtworks.xstream.mapper.ArrayMapper.realClass(ArrayMapper.java:74)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
at com.thoughtworks.xstream.mapper.SecurityMapper.realClass(SecurityMapper.java:71)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
at hudson.util.XStream2$CompatibilityMapper.realClass(XStream2.java:452)
at hudson.util.xstream.MapperDelegate.realClass(MapperDelegate.java:46)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
at com.thoughtworks.xstream.mapper.CachingMapper.realClass(CachingMapper.java:47)
at hudson.util.RobustReflectionConverter.determineType(RobustReflectionConverter.java:521)
at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:346)
Caused: jenkins.util.xstream.CriticalXStreamException:
---- Debugging information ----
cause-exception : com.thoughtworks.xstream.mapper.CannotResolveClassException
cause-message : hudson.security.ProjectMatrixAuthorizationStrategy
class : hudson.model.Hudson
required-type : hudson.model.Hudson
converter-type : hudson.util.RobustReflectionConverter
path : /hudson/authorizationStrategy
line number : 11
version : 2.422

at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:381)
at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:289)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:52)
at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:136)
at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1464)
at hudson.util.XStream2.unmarshal(XStream2.java:230)
at hudson.util.XStream2.unmarshal(XStream2.java:201)
at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1441)
at hudson.XmlFile.unmarshal(XmlFile.java:196)

Caused: java.io.IOException: Unable to read /opt/jenkins/config.xml
at hudson.XmlFile.unmarshal(XmlFile.java:199)
at hudson.XmlFile.unmarshal(XmlFile.java:179)
at jenkins.model.Jenkins.loadConfig(Jenkins.java:3379)
at jenkins.model.Jenkins$12.run(Jenkins.java:3491)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:177)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)
at jenkins.model.Jenkins$5.runTask(Jenkins.java:1170)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused: org.jvnet.hudson.reactor.ReactorException
at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:290)
at jenkins.InitReactorRunner.run(InitReactorRunner.java:49)
at jenkins.model.Jenkins.executeReactor(Jenkins.java:1205)
at jenkins.model.Jenkins.(Jenkins.java:992)
at hudson.model.Hudson.(Hudson.java:86)
at hudson.model.Hudson.(Hudson.java:82)
at hudson.WebAppMain$3.run(WebAppMain.java:247)
Caused: hudson.util.HudsonFailedToLoad
at hudson.WebAppMain$3.run(WebAppMain.java:264)

I figure I might need to upgrade plugins, but there’s no interface to do this. Is there a step I’m missing?

Cheers,

Brendan

Is /opt/jenkins set as your JENKINS_HOME? Does the /opt/jenkins/config.xml exist?

You should analyze the complete systemd log with
journalctl -u jenkins and look at the very first error. This is the root cause for all following problems in loading the plugins.

Thanks Alex and Markus. The first output was from journalctl -u jenkins and the config.xml did exist. In the end I decided to roll back to my snapshot (no time to troubleshoot with developers needing to use the server!) and I will regroup and attempt again.

I notice that quite a few of the plugins that are installed need updates and many of them are saying that they won’t after update due to version dependencies, so I might raise a separate question about how best to handle that - my original post in this one will probably muddy the waters too much.



If you still have the image.
Check permissions on /opt/config/jenkins.xml including if it's a mount point and that mount point is writeable.

Also check that Jenkins is run by user "jenkins and that it's readable Nay writeable by that user


Caused: java.io.IOException: Unable to read /opt/jenkins/config.xml
If you still have the image.
Check the permissions on /opt/je

Caused: java.io.IOException: Unable to read /opt/jenkins/config.xml