Jenkins version Upgrade in RHEL 8.5

[Spandana]

Hi Team,

I want to upgrade Jenkins version from 2.319.1 to latest LTS version (please provide the details to which version I can upgrade to).

• To upgrade to the latest version, Do I need to follow any levels to reach final version or directly can we shift from current version to latest version?
• Do we need to upgrade plugins as well after the Jenkins version upgrade or what steps should be followed for plugins?
• Please provide the steps to follow for Jenkins version upgrade as this is using by multiple product team, we want to make sure to follow best practices
• Provide any suggestions

Jenkins setup:

Jenkins: 2.319.1

OS: Linux - 4.18.0-348.7.1.el8_5.x86_64 Java: 1.8.0_412 - Red Hat, Inc. (OpenJDK 64-Bit Server VM) --- Office-365-Connector:4.17.0 ace-editor:1.1 amazon-ecr:1.73.v741d474abe74 ant:1.13 antisamy-markup-formatter:2.6 apache-httpcomponents-client-4-api:4.5.13-1.0 appscan:1.0.14 authentication-tokens:1.4 aws-credentials:189.v3551d5642995 aws-java-sdk:1.12.148-310.v5e3b_c2681d79 aws-java-sdk-cloudformation:1.12.148-310.v5e3b_c2681d79 aws-java-sdk-codebuild:1.12.148-310.v5e3b_c2681d79 aws-java-sdk-ec2:1.12.148-310.v5e3b_c2681d79 aws-java-sdk-ecr:1.12.148-310.v5e3b_c2681d79 aws-java-sdk-ecs:1.12.148-310.v5e3b_c2681d79 aws-java-sdk-elasticbeanstalk:1.12.148-310.v5e3b_c2681d79 aws-java-sdk-iam:1.12.148-310.v5e3b_c2681d79 aws-java-sdk-logs:1.12.148-310.v5e3b_c2681d79 aws-java-sdk-minimal:1.12.148-310.v5e3b_c2681d79 aws-java-sdk-ssm:1.12.148-310.v5e3b_c2681d79 blueocean:1.25.5 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.25.5 blueocean-commons:1.25.5 blueocean-config:1.25.5 blueocean-core-js:1.25.5 blueocean-dashboard:1.25.5 blueocean-display-url:2.4.1 blueocean-events:1.25.5 blueocean-git-pipeline:1.25.5 blueocean-github-pipeline:1.25.5 blueocean-i18n:1.25.5 blueocean-jwt:1.25.5 blueocean-personalization:1.25.5 blueocean-pipeline-api-impl:1.25.5 blueocean-pipeline-editor:1.25.5 blueocean-pipeline-scm-api:1.25.5 blueocean-rest:1.25.5 blueocean-rest-impl:1.25.5 blueocean-web:1.25.5 bootstrap4-api:4.6.0-3 bootstrap5-api:5.1.3-6 bouncycastle-api:2.25 branch-api:2.1046.v0ca_37783ecc5 build-timeout:1.20 caffeine-api:2.9.2-29.v717aac953ff3 checks-api:1.7.2 cloudbees-bitbucket-branch-source:773.v4b_9b_005b_562b_ cloudbees-folder:6.729.v2b_9d1a_74d673 cobertura:1.17 code-coverage-api:2.0.4 command-launcher:1.6 config-file-provider:3.8.2 copyartifact:1.46.4 credentials:1087.1089.v2f1b_9a_b_040e4 credentials-binding:1.27.1 data-tables-api:1.11.4-4 display-url-api:2.3.5 docker-commons:1.19 docker-java-api:3.2.13-37.vf3411c9828b9 docker-plugin:1.2.9 docker-workflow:1.28 durable-task:493.v195aefbb0ff2 echarts-api:5.3.2-1 email-ext:2.86 favorite:2.3.3.1 file-operations:1.11 font-awesome-api:6.0.0-1 forensics-api:1.13.0 git:4.11.3 git-client:3.11.0 git-server:1.10 github:1.34.1 github-api:1.301-378.v9807bd746da5 github-branch-source:2.11.4 gradle:1.37.1 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 htmlpublisher:1.30 jackson2-api:2.13.3-285.vc03c0256d517 javadoc:226.v71211feb_e7e9 javax-activation-api:1.2.0-3 javax-mail-api:1.6.2-5 jaxb:2.3.6-1 jdk-tool:1.5 jenkins-design-language:1.25.5 jjwt-api:0.11.2-9.c8b45b8bb173 jquery3-api:3.6.0-3 jsch:0.1.55.2 junit:1.53 ldap:2.7 lockable-resources:2.13 login-theme:1.1 mailer:408.vd726a_1130320 matrix-auth:3.0 matrix-project:772.v494f19991984 maven-plugin:3.16 miniorange-saml-sp:1.0.5 momentjs:1.1.1 nodejs:1.5.1 okhttp-api:4.9.3-105.vb96869f8ac3a pam-auth:1.6.1 pipeline-aws:1.43 pipeline-build-step:2.15 pipeline-github-lib:1.0 pipeline-graph-analysis:188.v3a01e7973f2c pipeline-input-step:427.va6441fa17010 pipeline-milestone-step:1.3.2 pipeline-model-api:1.9.3 pipeline-model-definition:1.9.3 pipeline-model-extensions:1.9.3 pipeline-rest-api:2.20 pipeline-stage-step:291.vf0a8a7aeeb50 pipeline-stage-tags-metadata:1.9.3 pipeline-stage-view:2.20 plain-credentials:1.8 plugin-util-api:2.16.0 popper-api:1.16.1-2 popper2-api:2.11.5-1 pubsub-light:1.16 resource-disposer:0.17 role-strategy:3.2.0 s3:0.12.1 saml:2.1.1-275.va_5718591a_999 scm-api:608.vfa_f971c5a_a_e9 script-security:1138.v8e727069a_025 snakeyaml-api:1.29.1 sse-gateway:1.25 ssh-credentials:1.19 ssh-slaves:1.33.0 sshd:3.1.0 structs:318.va_f3ccb_729b_71 terraform:1.0.10 timestamper:1.15 token-macro:267.vcdaea6462991 trilead-api:1.0.13 variant:1.4 workflow-aggregator:2.6 workflow-api:1153.vb_912c0e47fb_a_ workflow-basic-steps:2.24 workflow-cps:2660.vb_c0412dc4e6d workflow-cps-global-lib:552.vd9cc05b8a2e1 workflow-durable-task-step:1112.vda00e6febcc1 workflow-job:1145.v7f2433caa07f workflow-multibranch:716.vc692a_e52371b_ workflow-scm-step:2.13 workflow-step-api:625.vd896b_f445a_f8 workflow-support:813.vb_d7c3d2984a_0 ws-cleanup:0.40

[MarkEWaite (Mark Waite)]

@Spandana

I want to upgrade Jenkins version from 2.319.1 to latest LTS version

• Do I need to follow any levels to reach final version or directly can we shift from current version to latest version?

You’re making a good choice to upgrade. There are multiple critical security vulnerabilities in your current Jenkins version. You should also upgrade your operating system, since Red Hat Enterprise Linux 8.10 is the currently released version of Red Hat Enterprise Linux 8.

That’s a very large change in Jenkins versions. The Jenkins project recommends that users upgrade LTS versions every month. You’ve not upgraded in 32 months. You should plan to spend time evaluating and testing the upgrade in a copy of your environment before you attempt it on the production installation. You’re paying 32 months of upgrade debt.

Here are some of the answers that have been previously given. Refer to them for more details

Best Jenkins Upgrade Path

Depends greatly on your definition of best. Most reliable: Upgrade all the plugins you can Upgrade to next jenkins stable release follow Jenkins LTS Upgrade Guide Upgrade all plugins you can Repeat until you are at the most recent version Easiest Shortest: YOLO Make a backup of your config. Install latest into a new directory/server/etc Copy jobs from old one Startup jenkins Install all the plugins you used to have. There’s no quick and easy and reliable way to upgrade 251->387 which is …

Recommended Upgrade Path Jenkins -?

That is a very large increment of Jenkins versions, spanning multiple significant changes in the Jenkins project. Information about those significant changes are in the LTS changelog and in the upgrade guides. Read the changelog and the upgrade guide very carefully. Your organization has chosen to not upgrade Jenkins for 3 years. That has created some “upgrade debt” that you’ll now need to repay. @mwp565733 shared experience with that type of upgrade in a blog post. It is worthwhile readi…

The Jenkins LTS upgrade guides describe the changes from your current version to the most recent Jenkins LTS. Read the upgrade guides to identify areas that may be important to your installation.

Some of the key changes between those two versions include:

• Uses Java 17 or Java 21 instead of Java 8
• Fixes for critical security vulnerabilities
• Includes multiple user interface improvements
• Removes multiple outdated components

The Jenkins LTS changelog describes the details of each of the releases between your current release and the most recent release. Read the changelogs for each of the intervening releases and identify areas that may be important to your organization.

@mwp565733 shared an upgrade experience in a Jenkins blog post. Read the blog post:

Back of the Napkin Guide to Updating Jenkins, for the uninitiated

@Spandana

• Do we need to upgrade plugins as well after the Jenkins version upgrade or what steps should be followed for plugins?

Yes, you’ll need to install the most recent version of the plugins you currently have installed. This is also a time to remove deprecated plugins and to remove plugins with known security vulnerabilities.

The Jenkins plugins site now includes a plugin health score for each plugin. This is a good time to evaluate if there are plugins that should be removed because they are not being maintained or because they are no longer being used.

@Spandana

• Please provide the steps to follow for Jenkins version upgrade as this is using by multiple product team, we want to make sure to follow best practices

If multiple product teams are using the system, then I assume they care very much that the upgrade runs smoothly with very little downtime. If so, then follow the recommendations in the mentioned posts by creating a backup of the current installation and performing the upgrade on that backup. Confirm the upgrade works on the backup in all the cases that are important to your organization.

[MarkEWaite (Mark Waite)]

@aadams26

Do you recommend that the OS is upgraded first or Jenkins?

I would install a new machine with the new operating system then install the old Jenkins from an operating system package, restore the Jenkins backup from the old machine, then upgrade the Jenkins version on the new machine.

[Spandana]

Hi Mark,

Thank you for the details

As suggested, I’ve restored my Jenkins data to new machine with same OS version RHEL 8.5 and started testing on it.
As a first step, as soon as I upgraded my Java version from 11 to 17, My Jenkins(2.3191v) is not accessible.

And then I tried to upgrade Jenkins version to 2.479.1 on java 17, but still facing same issue. Also I tried to upgrade Jenkins version on Java 11, still no luck.

please guide me.

Regards,
Spandana K

[MarkEWaite (Mark Waite)]

@Spandana

As a first step, as soon as I upgraded my Java version from 11 to 17, My Jenkins(2.319.1) is not accessible.

That is expected. The Java support policy notes that 2.346.1 is the first Jenkins LTS to support Java 17. You can’t run Jenkins 2.319.1 with Java 17 or Java 21.

@Spandana

And then I tried to upgrade Jenkins version to 2.479.1 on java 17, but still facing same issue.

You’ll need to provide more detail that describes what you mean when you say that you are facing the “same issue”. What issue are you facing? If you’re trying to run Jenkins 2.479.1 with Java 11, that won’t work. Jenkins 2.479.1 requires Java 17 or Java 21.

@Spandana

Also I tried to upgrade Jenkins version on Java 11, still no luck.

Saying “still no luck” doesn’t allow others to help you. What Jenkins version were you trying to run with Java 11? Does that Jenkins version support Java 11? If it does, what were the failure messages?

@Spandana

please guide me.

Did you read each of the pages that I included in the earlier response? Did you decide which upgrade path you will attempt, either a single big step or a series of smaller steps? Did you create a separate machine where you can test drive the upgrade path you have chosen?

[Spandana]

As mentioned earlier, Currently, I am using Jenkins version 2.319.1 on RHEL 8.5 with Java 11 and Nginx 1.14.1.

Following your suggestion, I restored the Jenkins production data to the new VM to proceed with actual testing.

In the new VM, I upgraded the Java version from 11 to 17, as Jenkins 2.479.1 requires Java 17. I then upgraded Jenkins to version 2.479.1 by replacing /usr/lib/jenkins with 2.479.1 jenkins.war, and restarted Jenkins.

After these steps, the Jenkins URL is not accessible, showing a 501 Bad Gateway error. The Jenkins status displays the following error:

2024-12-01 14:42:42.434+0000 [id=215] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started DockerContainerWatchdog Asynchronous Periodic Work
2024-12-01 14:42:42.434+0000 [id=215] INFO c.n.j.p.d.DockerContainerWatchdog#execute: Docker Container Watchdog has been triggered
2024-12-01 14:42:42.434+0000 [id=215] INFO c.n.j.p.d.DockerContainerWatchdog$Statistics#writeStatisticsToLog: Watchdog Statistics: Number of overall executions: 12, Executions with processing timeout: 0, Containers removed gracefully: 0, Containers removed with force: 0, Containers removal failed: 0, Nodes removed successfully: 0, Nodes removal failed: 0, Container removal average duration (gracefully): 0 ms, Container removal average duration (force): 0 ms, Average overall runtime of watchdog: 0 ms, Average runtime of container retrieval: 0 ms
2024-12-01 14:42:42.435+0000 [id=215] INFO c.n.j.p.d.DockerContainerWatchdog#loadNodeMap: We currently have 0 nodes assigned to this Jenkins instance, which we will check
2024-12-01 14:42:42.435+0000 [id=215] INFO c.n.j.p.d.DockerContainerWatchdog#execute: Docker Container Watchdog check has been completed
2024-12-01 14:42:42.435+0000 [id=215] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished DockerContainerWatchdog Asynchronous Periodic Work. 1 ms
2024-12-01 14:44:11.069+0000 [id=48] INFO winstone.Logger#logInternal: JVM is terminating. Shutting down Jetty

=======================================
Here are the steps I am taking:

1. Upgrade Java from version 11 to 17.
2. Upgrade Jenkins from version 2.319.1 to 2.479.1.
3. Upgrade Nginx from version 1.14.1 to 1.18.0.

Regarding Plugins: Most of the plugins on my Jenkins 2.319.1 are displaying the following warning message.
Warning: This plugin is built for Jenkins 2.361.4 or newer. Jenkins will refuse to load this plugin if installed.
Warning: This plugin is built for Jenkins 2.401.3 or newer. Jenkins will refuse to load this plugin if installed.

I suspect that the plugins are refusing to access the Jenkins URL due to the Jenkins version upgrade. If this is the case, what steps should I follow to ensure compatibility between the Jenkins version and the plugins, so that I can access Jenkins without any issues?

[Spandana]

Hi Mark,

Please suggest me on this
This is the first time I’m working on Jenkins version upgradation and wanted to know the detailed steps that I need to follow here.

Regards,
Spandana K

[MarkEWaite (Mark Waite)]

@Spandana

This is the first time I’m working on Jenkins version upgradation and wanted to know the detailed steps that I need to follow here.

The detailed steps are provided in the pages that I linked earlier. Read those pages and then choose your upgrade path based on those pages.

[Spandana]

Definitely,

Also can you please provide the details on the error I’m getting while accessing Jenkins URL

In the new VM, I upgraded the Java version from 11 to 17, as Jenkins 2.479.1 requires Java 17. I then upgraded Jenkins to version 2.479.1 by replacing /usr/lib/jenkins with 2.479.1 jenkins.war, and restarted Jenkins.

After these steps, the Jenkins URL is not accessible, showing a 501 Bad Gateway error. The Jenkins status displays the following error:

2024-12-01 14:42:42.434+0000 [id=215] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started DockerContainerWatchdog Asynchronous Periodic Work
2024-12-01 14:42:42.434+0000 [id=215] INFO c.n.j.p.d.DockerContainerWatchdog#execute: Docker Container Watchdog has been triggered
2024-12-01 14:42:42.434+0000 [id=215] INFO c.n.j.p.d.DockerContainerWatchdog$Statistics#writeStatisticsToLog: Watchdog Statistics: Number of overall executions: 12, Executions with processing timeout: 0, Containers removed gracefully: 0, Containers removed with force: 0, Containers removal failed: 0, Nodes removed successfully: 0, Nodes removal failed: 0, Container removal average duration (gracefully): 0 ms, Container removal average duration (force): 0 ms, Average overall runtime of watchdog: 0 ms, Average runtime of container retrieval: 0 ms
2024-12-01 14:42:42.435+0000 [id=215] INFO c.n.j.p.d.DockerContainerWatchdog#loadNodeMap: We currently have 0 nodes assigned to this Jenkins instance, which we will check
2024-12-01 14:42:42.435+0000 [id=215] INFO c.n.j.p.d.DockerContainerWatchdog#execute: Docker Container Watchdog check has been completed
2024-12-01 14:42:42.435+0000 [id=215] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished DockerContainerWatchdog Asynchronous Periodic Work. 1 ms
2024-12-01 14:44:11.069+0000 [id=48] INFO winstone.Logger#logInternal: JVM is terminating. Shutting down Jetty

=======================================

[MarkEWaite (Mark Waite)]

@Spandana

Also can you please provide the details on the error I’m getting while accessing Jenkins URL

No, I cannot. You’ve not provided enough information so that others can duplicate the issue that you are seeing.

The message “JVM is terminating” may indicate that there is an entry you should read in the systemd log. The “Managing systemd services” documentation says that you can read that log with the command:

journalctl -u jenkins