Help! Updating Jenkins failed. Need to rollback, but some plugins are not loading

Using Jenkins Ask a question
1

Jenkins setup:

Jenkins: 2.474
OS: Linux - 3.10.0-1160.119.1.el7.x86_64
Java: 21 - Oracle Corporation (Java HotSpot(TM) 64-Bit Server VM)
---
Parameterized-Remote-Trigger:3.2.0
adoptopenjdk:1.5
analysis-model-api:12.4.0
ansicolor:1.0.4
ant:511.v0a_a_1a_334f41b_
antexec:196.v306c59b_7ca_ef
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
apache-httpcomponents-client-5-api:5.3.1-110.v77252fb_d4da_5
asm-api:9.7-33.v4d23ef79fcc8
authentication-tokens:1.119.v50285141b_7e1
bitbucket-oauth:0.13
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.78.1-248.ve27176eb_46cb_
branch-api:2.1178.v969d9eb_c728e
build-with-parameters:76.v9382db_f78962
bulk-builder:1.5
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.2.0
cloudbees-folder:6.942.vb_43318a_156b_2
command-launcher:115.vd8b_301cc15d0
commons-lang3-api:3.16.0-82.ve2b_07d659d95
commons-text-api:1.12.0-129.v99a_50df237f7
conditional-buildstep:1.4.3
config-file-provider:973.vb_a_80ecb_9a_4d0
credentials:1371.vfee6b_095f0a_3
credentials-binding:681.vf91669a_32e45
data-tables-api:2.1.4-1
display-url-api:2.204.vf6fddd8a_8b_e9
docker-commons:443.v921729d5611d
durable-task:568.v8fb_5c57e8417
echarts-api:5.5.1-1
eddsa-api:0.3.0-4.v84c6f0f4969e
external-monitor-job:215.v2e88e894db_f8
font-awesome-api:6.6.0-1
forensics-api:2.5.0
git-client:5.0.0
git-server:126.v0d945d8d2b_39
github-api:1.321-475.vf7ef62885c83
gson-api:2.11.0-41.v019fcf6125dc
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
heavy-job:1.1
htmlpublisher:1.36
instance-identity:201.vd2a_b_5a_468a_a_6
ionicons-api:74.v93d5eb_813d5f
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javadoc:280.v050b_5c849f69
javax-activation-api:1.2.0-7
javax-mail-api:1.6.2-10
jaxb:2.3.9-1
jdk-tool:80.v8a_dee33ed6f0
jersey2-api:2.44-151.v6df377fff741
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
jnr-posix-api:3.1.19-2
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery3-api:3.7.1-2
jsch:0.2.16-86.v42e010d9484b_
json-api:20240303-41.v94e11e6de726
json-path-api:2.9.0-58.v62e3e85b_a_655
junit:1307.vdd5b_2646279e
keyboard-shortcuts-plugin:1.4
locale:519.v4e20f313cfa_f
matrix-auth:3.2.3
mina-sshd-api-common:2.13.2-125.v200281b_61d59
mina-sshd-api-core:2.13.2-125.v200281b_61d59
nodejs:1.6.2
okhttp-api:4.11.0-179.vdc1e64343d52
pam-auth:1.11
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-groovy-lib:730.ve57b_34648c63
pipeline-input-step:495.ve9c153f6067b_
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2205.vc9522a_9d5711
pipeline-model-extensions:2.2205.vc9522a_9d5711
pipeline-multibranch-defaults:2.1
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2205.vc9522a_9d5711
pipeline-stage-view:2.34
plain-credentials:183.va_de8f1dd5a_2b_
plugin-util-api:4.1.0
postbuild-task:1.9
preSCMbuildstep:71.v1f2990a_37e27
prism-api:1.29.0-17
pubsub-light:1.18
run-condition:1.7
sbt:81.vb_82499046630
scm-api:696.v778d637b_a_762
script-security:1367.vdf2fc45f229c
snakeyaml-api:2.2-121.v5a_68b_9300b_d4
sse-gateway:1.27
ssh-credentials:343.v884f71d78167
sshd:3.330.vc866a_8389b_58
structs:338.v848422169819
testng-plugin:835.v51ed3da_fcc35
thinBackup:2.1.1
timestamper:1.27
token-macro:400.v35420b_922dcb_
trilead-api:2.147.vb_73cc728a_32e
variant:60.v7290fc0eb_b_cd
warnings-ng:11.4.1
workflow-api:1336.vee415d95c521
workflow-cps:3946.v7935cb_edb_f82
workflow-durable-task-step:1371.vb_7cec8f3b_95e
workflow-job:1436.vfa_244484591f
workflow-multibranch:795.ve0cb_1f45ca_9a_
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:678.v3ee58b_469476
workflow-support:920.v59f71ce16f04
xvfb:1.2

I’m running Jenkins on Open Liberty web container.
I tried to upgrade Jenkins to the new Jakarta version. At first, I tested the setup in a fresh Open Liberty install and fresh Jenkins application, and it worked.
When upgrading the existing Jenkins application, it failed. Other web applications (war) upgraded well, but not Jenkins. It seems there are old plugins which are not compatible, and so the start page cannot display the Jenkins panel, and I cannot access anything.
So far, I have to roll back to v2474. Jenkins starts and I have access to its features. But it could not load some plugins. It does not give an option to downgrade or to reinstall those plugins. Just reports that they could not be loaded (in Portuguese), such as:

Falhou para carregar: Jenkins Bitbucket Plugin (bitbucket 254.v9d8ec33a_7a_42)

  • Falha para carregar: Git plugin (git 5.4.1)

Falhou para carregar: Build Blocker Plugin (build-blocker-plugin 166.vc82fc20b_a_ed6)

  • Falha para carregar: Matrix Project Plugin (matrix-project 840.v812f627cb_578)

Falhou para carregar: Git Changelog (git-changelog 3.38)

  • Falha para carregar: Git plugin (git 5.4.1)

It seems the main plugins which could not be loaded are:

Matrix Project Plugin (matrix-project 840.v812f627cb_578)
Jenkins Mailer Plugin (mailer 489.vd4b_25144138f)
Git plugin (git 5.4.1)

The main message says that I have to upgrade Jenkins to v2479 or above.
But I cannot do that now.

My question is: how can I get the previous versions of those plugins and install tem back into Jenkins v2474?

Thanks in advance.

2

This is a snippet of the stack trace of the error which happens a lot in the UPGRADED install.

java.lang.RuntimeException: SRV.8.2: RequestWrapper objects must extend ServletRequestWrapper or HttpServletRequestWrapper
	at com.ibm.wsspi.webcontainer.util.ServletUtil.unwrapRequest(ServletUtil.java:91)
	at com.ibm.wsspi.webcontainer.util.ServletUtil.unwrapRequest(ServletUtil.java:63)
	at com.ibm.wsspi.webcontainer.util.ServletUtil.unwrapRequest(ServletUtil.java:58)
	at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:449)
	at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:197)
	at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:100)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:204)
	at io.jenkins.servlet.FilterChainWrapper$2.doFilter(FilterChainWrapper.java:53)
	at PluginClassLoader for sse-gateway//org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
	at io.jenkins.servlet.FilterWrapper$1.doFilter(FilterWrapper.java:42)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:201)
	at io.jenkins.servlet.FilterChainWrapper$2.doFilter(FilterChainWrapper.java:53)
	at PluginClassLoader for locale//hudson.plugins.locale.LocaleFilter.doFilter(LocaleFilter.java:64)
	at io.jenkins.servlet.FilterWrapper$1.doFilter(FilterWrapper.java:42)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:201)
	at jenkins.util.HttpServletFilter$1.doFilter(HttpServletFilter.java:77)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:201)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:207)
	at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:203

It seems to me that some plugin or plugins are not compatible and must be upgraded as well.
I wonder which exactly are those plugins. And I wonder whether upgrading all plugins before upgrading Jenkins would solve it, or it would screw the installation altogether.

3

Good news. In the plugins folder I found out that the mailer and the matrix project plugins where updated without my consent (!). I reverted them to the previous file (bak) and now Jenkins is fully functional again.

I’m still wondering whether I should update all plugins BEFORE update Jenkins.

4

The Jenkins project does not test the Open Liberty web container. You’re running in an environment that the servlet container support policy notes is either level 2 (patches accepted) or level 3 (unsupported).

Jenkins 2.475 includes multiple upgrades and significant changes, including:

  • upgrade Spring Framework from 5.3.39 to 6.1.12
  • upgrade Spring Security from 5.8.14 to 6.3.3
  • upgrade Java EE from 8 to 9
  • upgrade to Jetty 12 EE 9

The Jenkins 2.475 changelog](Changelog) changelog says:

Users of third-party servlet containers must upgrade the servlet container to an EE 9 version in accordance with the Jenkins Servlet Container Support Policy

The general guidance for Jenkins upgrades is:

  • Backup the system
  • Upgrade the plugins
  • Upgrade Jenkins
  • Upgrade the plugins again

With Jenkins 2.475 there are additional instructions that if using the LDAP plugin, CAS plugin, Reverse Proxy Auth plugin, or Windows Negotiate SSO plugin, then those plugins must be upgraded while the Jenkins controller is stopped. More details are in the 2.479.1 LTS upgrade guide.

The procedure that you are using to upgrade plugins seems to have a flaw. The Jenkins plugin manager and the Jenkins plugin installation manager tool will not offer plugins that are incompatible with your versions. Use them to update plugins rather than just downloading the most recent release of each plugin.

5

Thanks.
I’ve always upgraded plugins after upgrading Jenkins. I didn’t know I should upgrade them before.
Also, I wasn’t aware of that document: 2.479.1 LTS upgrade guide. I couldn’t find it in the changelog.
I will take some time to follow that document to upgrade the current install to the new Jakarta version. I’ve already get rid of some unused plugins, and maybe the next time it will work better. I’ll let you know the results.