Subject: Jenkins 2.528.3 Upgrade Fails on Azure VM but Works On-Prem – Plugin & ClassNotFound Issues (Matrix-Auth / CloudBees Folder)
Description:
Hi Team,
I am facing an issue while upgrading Jenkins from 2.440.3 to 2.528.3 on a RHEL 8 Azure VM, whereas the same upgrade works successfully on an on-prem VM.
What works
On-prem RHEL 8 VM:
Upgrade completes successfully
Jenkins starts (with some plugin warnings, but service is up)
What fails
Azure RHEL 8 VM:
Upgrade completes, but Jenkins service fails to start
Jenkins initialization is blocked due to plugin errors
You will need to look at the complete log of the failing instance. The first error where it fails to load a plugin is the problem. Maybe that is the only problem and all others are just follow up problems.
Did you follow the upgrade guides? Did you update all plugins before upgrading Jenkins?
Sometimes it is necessary to update a single plugin in lockstep with upgrading Jenkins, the upgrade guide usually points to such requirements.
When you say instances are identical, are you 100% sure? Maybe there was a mismatch in plugin versions.
We have our current Jenkins (v2.440.3) installed using RPM package. We tried upgrading it to 2.528.3 using rpm again, but Jenkins service did not start. So, we installed Jenkins on non-prod VM and copied plugins and jobs to that VM to try and troubleshoot the issue.
There we saw that after upgrading there were some issues however, it did not stop Jenkins service from getting up. The difference we observed was the config.xml.
On prod VM, config.xml was of about 15000 lines with lots of plugin and other details but on non-prod there were only 58 lines as a fresh config.xml was used. So, I want to understand the right approach to upgrade the Jenkins.
Is updating all plugins suggested in Updates UI page of Jenkins before upgrading core is a good option?
2.528.3 requires at least Java 17 so make sure you use that. Better to use to Java 21 already now.
Read the upgrade guides for 2.528 and all intermediate releases. Especially the upgrade to 2.479.3 might contain things to consider as this brought the need to use Java 17 and changed Jenkins from EE8 to EE9
For the plugins to load it doesn’t matter what it is in the config.xml. the config.xml contains user permissions, view configs and cloud configs and some other config stuff from core. Plugins (except for auth and permissions) normally have their config in separate xml files in JENKINS_HOME normally
How I would do it:
Take a backup of JENKINS_HOME before doing anything
Create a test instance that has the same Jenkins and same plugins as prod instance, it should start without any problems. Maybe just copy the complete JENKINS_HOME from the prod instance except for the jobs (so the 2 instances don’t interfere by both starting to vote on scm changes). Also make sure that agents work in different directories. Create some tests jobs that do the same things like you expect on the prod instance
stop Jenkins test instance
update Jenkins (via rpm)
use plugin manager tool to update all plugins
start Jenkins
Jenkins should now start without any problems normally.
Now you can do the same on the prod instance (after you have taken a backup):
I am going through jenkins-plugin-cli tool and want to know if I only provide list of plugin names in plugins file, will it attempt to upgrade it to latest versions? Or it will selectively install the plugins which are compatible with each other as well as core?
The tool will only install plugins that are compatible with the given Jenkins version. At least when your Jenkins version is not too old (not more than approx. 1 year)
Thanks for your inputs. I was able to upgrade the Jenkins and plugin-cli helped tremendously to get there. For a long term approach, do you think it is better to have blue-green Jenkins servers load balanced through dns to minimize the downtime? Has anyone done that before? Any tips? Thanks.
Afaik Cloudbess has a High Availability solution for Jenkins but you have to pay for this.
Opensource Jenkins can’t be run behind a load balancer. Jenkins keeps state in memory and persists it to disk, but reads the disk only at startup or when told to do so. So when 2 instances would run on the same file system (e.g. via a NFS share) you would get concurrent writes.
My recommendation is to run Jenkins in a docker container. Create you own image with Jenkins and plugins baked in. That way when you update the image you can easily start the image on a separate machine to see whether it starts properly. I’m using this approach for 10 years now and never had any issues during upgrades.
What works
What fails