New Install versus Upgrade for instance many versions behind and with many old plugins

We have been using Jenkins since 2016/2017. Until a couple of years ago, no one was responsible for trying to keep the system updated. That has fallen on me now. The previous administrator installed a bunch of plugins that are way behind or deprecated. I have gotten our instance up to LTS 2.303.3 so far.

I am wondering though if I should consider a different approach. What if I installed the newest LTS version on a new server? How would I migrate the jobs and credentials to the new system? I think I would need to do the configuration (under Configure System, Global Tool Configuration, Configure Global Security) manually. Is there a list of “recommended” plugins to start off with and then I could just add others as needed? Many of the plugins we have currently we added because they were not part of Core originally.

I know this is a very open-ended question/topic. Any helpful suggestions of how to best modernize our Jenkins environment would be appreciated.

I’m currently looking into upgrading a 2.60.3 Jenkins to latest LTS. It’s not yet finished.
What I did so far was the following:

Starting point:
Jenkins 2.60.3 with all plugins on the latest state compatible with this version
Jenkins running with tomcat

Preparation:

  • installed java17
  • installed jenkins rpm package
  • configured the systemd jenkins service with required startup parameters (java options (max memory, gc settings, jenkins specific properties), keystore, keystore password, access logging, jenkins_home)
  1. I copied the complete Jenkins home to a new directory
  2. created init.groovy script that will make the instance come up in shutdown mode
  3. created a file with all installed plugins (one per line without version number)
  4. With the plugin list and GitHub - jenkinsci/plugin-installation-manager-tool: Plugin Manager CLI tool for Jenkins I downloaded the latest version of all the plugins to a separate folder
  5. In the new jenkins location I removed everything in the plugins folder and copied all plugin.jpi files from step 4
  6. start Jenkins from the copied location (with a different port so it doesn’t interfere with the still running old instance)

The new Jenkins came up without any real problems. I was able to login, permissions where working all plugins could be loaded.
Agents couldn’t connect (only outbound ssh agents) because they where still trying to use java8, but that was expected. After installing java17 on the agents (and altering the root fs for the agents)
Of course there are tons of messages from Manage Old data all about unreadable data. But this is expected as well. Almost all messages are from older runs about IncompatibleClassChangeError: class com.google.common.collect.EmptyImmutableSortedSet overrides final method com.google.common.collect.ImmutableCollection.toArray()[Ljava/lang/Object;
That is usually uncritical as it might only mean in worst case we can’t properly read the status of old runs. But so far I didn’t see any problems in accessing old runs.

post upgrade work:
I visited plugin manager and uninstalled all deprecated plugins (e.g. ace-editor, handlebars, windows-slaves and some more)
Newer authorization strategies distinguish between permissions for users and groups so I had to act here.
Installed some new plugins that I consider helpful
The email configuration needed some attention, I had to set the TLS flag (both on mailer and emailext), the emailext didn’t migrate the password to a credential so I had to create a new credential for it.

Open points:
convert some long runnings freestyle jobs (up to 1 day runtime) to pipeline jobs so future updates do not kill the jobs

Maybe it was quite easy as the Jenkins doesn’t do any fancy things. Mostly freestyle jobs that execute some shell step and some have a few postbuild actions and a handful of pipeline jobs.

I’ve chosen this way because when I set up a complete new Jenkins, it means I need to reenter all the credentials and make sure to use exactly the same id. Otherwise when you copy your jobs folder, any job that makes use of credentials will fail as it will not find the credentials it needs.

1 Like

This blog post is also worth looking at: Back of the Napkin Guide to Updating Jenkins, for the uninitiated

Thanks a lot Markus, I think your answer will help lots of users.
If you feel like it, another blog post about your recent findings/modus operandi would be more visible than this thread.
My 0.02$…

Thank you so much for the detailed response. I will review this further and may have a few questions. I think one of my frustrations is that so many plugins were installed here. It would be great if Jenkins would only install the latest plugin versions compatible with the Jenkins version. Instead, I have at least 100 plugins to go through and determine what version of each one is compatible with the current version to upgrade those (manually, using the CLI feature I think) and then do the same for each subsequent version (I am using LTS versions).
One issue I found when I moved from 2.263 to 2.277, I tried to go in and uninstall the plugins identified as deprecated, however, I was unable to for a few because other plugins that were installed required some of the deprecated ones.
Thanks again for your reply. I will be reviewing it to see if I can determine my path forward!

Usually plugins work with any version that is newer than the version they were compiled against. Of course there might be exceptions.

The update center supports only Jenkins version that are not older than 1 years approx. So when you update an older version via the UI it very likely happens that you get plugins installed that do not work with that version. In that case it might be good to write your own script that fetches compatible version.

https://updates.jenkins.io/current/plugin-versions.json delivers a big json, that contains all versions of all plugins and for each version the required core and the dependencies.

1 Like