I’m running Jenkins on Open Liberty web container.
I tried to upgrade Jenkins to the new Jakarta version. At first, I tested the setup in a fresh Open Liberty install and fresh Jenkins application, and it worked.
When upgrading the existing Jenkins application, it failed. Other web applications (war) upgraded well, but not Jenkins. It seems there are old plugins which are not compatible, and so the start page cannot display the Jenkins panel, and I cannot access anything.
So far, I have to roll back to v2474. Jenkins starts and I have access to its features. But it could not load some plugins. It does not give an option to downgrade or to reinstall those plugins. Just reports that they could not be loaded (in Portuguese), such as:
Falhou para carregar: Jenkins Bitbucket Plugin (bitbucket 254.v9d8ec33a_7a_42)
Falha para carregar: Git plugin (git 5.4.1)
Falhou para carregar: Build Blocker Plugin (build-blocker-plugin 166.vc82fc20b_a_ed6)
Falha para carregar: Matrix Project Plugin (matrix-project 840.v812f627cb_578)
Falhou para carregar: Git Changelog (git-changelog 3.38)
Falha para carregar: Git plugin (git 5.4.1)
It seems the main plugins which could not be loaded are:
This is a snippet of the stack trace of the error which happens a lot in the UPGRADED install.
java.lang.RuntimeException: SRV.8.2: RequestWrapper objects must extend ServletRequestWrapper or HttpServletRequestWrapper
at com.ibm.wsspi.webcontainer.util.ServletUtil.unwrapRequest(ServletUtil.java:91)
at com.ibm.wsspi.webcontainer.util.ServletUtil.unwrapRequest(ServletUtil.java:63)
at com.ibm.wsspi.webcontainer.util.ServletUtil.unwrapRequest(ServletUtil.java:58)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:449)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:197)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:100)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:204)
at io.jenkins.servlet.FilterChainWrapper$2.doFilter(FilterChainWrapper.java:53)
at PluginClassLoader for sse-gateway//org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
at io.jenkins.servlet.FilterWrapper$1.doFilter(FilterWrapper.java:42)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:201)
at io.jenkins.servlet.FilterChainWrapper$2.doFilter(FilterChainWrapper.java:53)
at PluginClassLoader for locale//hudson.plugins.locale.LocaleFilter.doFilter(LocaleFilter.java:64)
at io.jenkins.servlet.FilterWrapper$1.doFilter(FilterWrapper.java:42)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:201)
at jenkins.util.HttpServletFilter$1.doFilter(HttpServletFilter.java:77)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:201)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:207)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:203
It seems to me that some plugin or plugins are not compatible and must be upgraded as well.
I wonder which exactly are those plugins. And I wonder whether upgrading all plugins before upgrading Jenkins would solve it, or it would screw the installation altogether.
Good news. In the plugins folder I found out that the mailer and the matrix project plugins where updated without my consent (!). I reverted them to the previous file (bak) and now Jenkins is fully functional again.
I’m still wondering whether I should update all plugins BEFORE update Jenkins.
The Jenkins project does not test the Open Liberty web container. You’re running in an environment that the servlet container support policy notes is either level 2 (patches accepted) or level 3 (unsupported).
Jenkins 2.475 includes multiple upgrades and significant changes, including:
upgrade Spring Framework from 5.3.39 to 6.1.12
upgrade Spring Security from 5.8.14 to 6.3.3
upgrade Java EE from 8 to 9
upgrade to Jetty 12 EE 9
The Jenkins 2.475 changelog](Changelog) changelog says:
Users of third-party servlet containers must upgrade the servlet container to an EE 9 version in accordance with the Jenkins Servlet Container Support Policy
The general guidance for Jenkins upgrades is:
Backup the system
Upgrade the plugins
Upgrade Jenkins
Upgrade the plugins again
With Jenkins 2.475 there are additional instructions that if using the LDAP plugin, CAS plugin, Reverse Proxy Auth plugin, or Windows Negotiate SSO plugin, then those plugins must be upgraded while the Jenkins controller is stopped. More details are in the 2.479.1 LTS upgrade guide.
The procedure that you are using to upgrade plugins seems to have a flaw. The Jenkins plugin manager and the Jenkins plugin installation manager tool will not offer plugins that are incompatible with your versions. Use them to update plugins rather than just downloading the most recent release of each plugin.
Thanks.
I’ve always upgraded plugins after upgrading Jenkins. I didn’t know I should upgrade them before.
Also, I wasn’t aware of that document: 2.479.1 LTS upgrade guide. I couldn’t find it in the changelog.
I will take some time to follow that document to upgrade the current install to the new Jakarta version. I’ve already get rid of some unused plugins, and maybe the next time it will work better. I’ll let you know the results.
I have news, though we are not ready yet.
I followed the upgrade steps, but still not working.
backup the system - OK.
upgrade the plugins - OK. The service is restarted and Jenkins is working.
upgrade Jenkins - The container is upgraded to EE 9 and jenkins.war is upgraded. The service fails with several instances of: SRV.8.2: RequestWrapper objects must extend ServletRequestWrapper or HttpServletRequestWrapper
I installed previously a clean jenkins.war (without plugins) inside this EE 9 container and it worked. So, the container must be OK. I suppose the container cannot start this application without having the plugins updated as well.
Since I cannot access the Jenkins plugin manager, I suppose I have to get the most recent release of each plugin manually. I will figure out how to do it.
Edit: I checked, and we don’t use some of those plugins which are reported as having to be updated manually.
I would like an advice to upgrade the plugins.
What is the best way to use Jenkins plugin installation manager tool to upgrade all plugins inside JENKINS_HOME?
I checked the documentation and it seems I have to manually craft a list of plugins to upgrade.
I found out I have to upgrade the mailer plugin manually, but it isn’t enough. There are more plugins that are not compatible. If I remove all plugins, Jenkins starts, but I cannot log in because it’s missing the matrix authorization plugin, which depends on a log of other plugins. After all, I have to upgrade all plugins at once - I suppose.
Create the plugins.txt file in the current installation with the results from “Manage Jenkins” → “Script Console” using the “How to report an issue” script
Solved.
I used the tool to get all the updated plugins. After many tests, I found the culprits: the Locale plugin and the Server Sent Events Gateway plugin.
I removed those plugins and now the Jenkins service runs fine.
Thanks for your support!
Though the service appears to be working, I just found out that the Bitbucket webhook is not working. The Bitbucket plugin must be updated. I just opened an issue for this.
For this reason I must revert the server update. I hope it works.