Jenkins setup: Jenkins 2.440.1 on Rocky Linux 9 with Java 17.0.11 within a docker container running on RHEL 7 VM.
Can we please consider having plugins more regulated to better match LTS versions?
Currently we are stuck on 2.440.1 because the plugins, especially pipeline plugins, keep being updated so frequently. There isn’t an LTS version of Jenkins I can update to and have a clean setup of plugins that we know will install cleanly and load all the existing jobs without issue. This really is a big issue for us due to the number of plugins we are using. Even if I carefully cherry pick which plugins I can update there are issues with loading existing jobs. It’s a very chaotic situation which seems to be in need of more regulation in the plugins release life cycle.
It would be great if plugin releases were modulated into more of a regular LTS (or quarterly?) release cycle and targeted a specific LTS version of Jenkins CLI. Otherwise I don’t know how we are going to get plugins updated.
I’ve also looked at non-LTS Jenkins 2.459 but upon updating what appears to be all the latest plugins the installation of plugins seems to be stalled here:
2024-06-03 18:30:08.868+0000 [id=192] INFO h.m.UpdateCenter$UpdateCenterConfiguration#download: Downloading json-api
Plugins define a specific version of Jenkins as a minimum requirement and the updatecenter in the UI will take care to only should you plugins that are compatible to your Jenkins version (unless you are on a more than 1 year old Jenkins version, which you aren’t).
Maybe you explain in more detail which problems you have. From the little you posted here it is hard to tell, e.g. how are you managing your plugins, do you create a docker image with plugins installed. which tools are you using to manage plugins.
Our entire /var/jenkins_home to a folder mounted to the host VM. This way we can do backups and replication to our DR VM(s) as required.
Here is a typical type of error from plugin version mismatches which have been happening during 2024 more than in previous years. This results in jobs which encounter an error preventing them from being loaded.
Below is an attempt to migration from LTS 2.440.1 to LTS 2.452.1 which did come up fine. But upon updating all plugins that could be updated the following issue occurred.
2024-06-04 13:04:08.515+0000 [id=40] SEVERE jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading item redacted.project.name
com.thoughtworks.xstream.mapper.CannotResolveClassException: org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject
at com.thoughtworks.xstream.mapper.DefaultMapper.realClass(DefaultMapper.java:81)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
at com.thoughtworks.xstream.mapper.DynamicProxyMapper.realClass(DynamicProxyMapper.java:55)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
at com.thoughtworks.xstream.mapper.PackageAliasingMapper.realClass(PackageAliasingMapper.java:88)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
at com.thoughtworks.xstream.mapper.ClassAliasingMapper.realClass(ClassAliasingMapper.java:79)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
at com.thoughtworks.xstream.mapper.ArrayMapper.realClass(ArrayMapper.java:74)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
at com.thoughtworks.xstream.mapper.SecurityMapper.realClass(SecurityMapper.java:71)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
at hudson.util.XStream2$CompatibilityMapper.realClass(XStream2.java:452)
at hudson.util.xstream.MapperDelegate.realClass(MapperDelegate.java:46)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
at com.thoughtworks.xstream.mapper.CachingMapper.realClass(CachingMapper.java:47)
at com.thoughtworks.xstream.core.util.HierarchicalStreams.readClassType(HierarchicalStreams.java:29)
at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:135)
at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1464)
at hudson.util.XStream2.unmarshal(XStream2.java:230)
at hudson.util.XStream2.unmarshal(XStream2.java:201)
at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1441)
at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1330)
at hudson.XmlFile.read(XmlFile.java:165)
We have also seen similar errors/incompatibilities in the groovy/pipeline plugins but do not have a recent stack trace on hand.
My suggestion was to bring plugins under LTS control/cadence. Alternately, if there was an ability for the Jenkins UI to interrogate plugin meta data for dependencies that could also flag which plugins are not matching.
I was under the impression that if a single plugin, say in the pipeline family had been updated to a newer version than the highest available LTS, then none of the pipeline plugins should be updated. This is based on previous similar errors like the above that I have seen when updating available pipeline plugins and skipping those pipeline plugins higher than the LTS version.
Yesterday I tested again taking all the recommended plugins that will work on our current LTS (2.440.1), including some in the pipeline section that were compatible and leaving out the ones in the pipeline section that are not. Yesterday when I tried this it worked to update plugins on LTS 2.440.1. But my recollection and experience is that this does not always work.
In my landscape we have a yaml file that contains all plugins and the version that should be installed. To update the versions we use a python script that reads the json from https://updates.jenkins.io/update-center.actual.json?version=2.452.1 which gives us all plugins in the latest version that is compatible to the given version (the python script exists since long time even before the above mentioned tool was introduced and is also capable to handle an internal update site with our inhouse plugins).
Ok, thanks for the tip on the command line tool, we didn’t know about this.
Before trying this tool out we were able copy up all available plugins from a local laptop install to update on our CICD VM running LTS 2.440.1 successfully as earlier indicated. Then we updated that CICD VM to LTS 2.452.1, so all good there.
However, running the tool raises more questions. As we had updated all plugins through the UI on the laptop sandbox, I expected the tool would not find anything to update. But it actually did, and not to versions that the Jenkins CLI showed as available or installable.
jtal@Johns-MacBook-Pro jenkins_aoln % ls -la plugins | grep jpi | grep 14:
236590 Jun 4 14:17 cloudbees-folder.jpi
4810142 Jun 4 14:17 delivery-pipeline-plugin.jpi
51225334 Jun 4 14:18 deployit-plugin.jpi
132210 Jun 4 14:17 parameterized-trigger.jpi
143643 Jun 4 14:17 workflow-job.jpi
88859 Jun 4 14:17 workflow-multibranch.jpi
In the UI for the workflow-job plugin prior to this run of the jenkins-plugin-manager it says this:
Pipeline: Job
Available Version 1426.v2ecb_a_a_42fd46
pipelineMiscellaneous
Defines a new job type for pipelines and provides their generic user interface.
Warning: This plugin is built for Jenkins 2.454 or newer. Jenkins will refuse to load this plugin if installed.
10 days ago Installed 1385.vb_58b_86ea_fff1
But when I look inside the META-INF/MANIFEST.MF of what is now installed in the plugins directory for workflow-job it gives this:
So we were at 1385.vb_58b_86ea_fff1 in the UI, and the UI said that latest was 1426.v2ecb_a_a_42fd46, but the jenkins-plugin-manager installed 1400.v7fd111b_ec82f
It is that the Plugin Manager in the UI is in need of rework or should not be used for updating plugins? I’m confused by this action of the jenkins-plugin-manager. The updated plugin does work with 2.452.1 so that’s great. But I never heard of this jenkins-plugin-manager and wondering why the UI isn’t smart enough to do what it just did? I’m guessing most of my issues this year has been trusting the Plugin Manager UI ?
What I’ve seen happening was that I had a test instance that I ran with current weekly, then I downgraded it to latest LTS. After that the plugin manager in the UI was showing me plugins that were incompatible. The reason was that Jenkins is caching the information about available plugins. So only after refreshing the information via the button in the upper right, it showed me the correct plugins.
It might also be worth checking the update center URL in Advanced settings, it should be https://updates.jenkins.io/update-center.json, Jenkins will append the version to the url to get the fitting plugins