SAML Plugin Failure Blocking Upgrade of 4000+ Jenkins Instances (2.462.3 to 2.479.3)

We are platform team from SAP and in our current setup Jenkins controller pods uses the image based on Jenkins version 2.462.3. The configuration specified in the Jenkins Setup section is common across all Jenkins controller pods, as they all use the same image built from version 2.462.3.

As part of our upgrade to Jenkins version 2.479.3, we built a new container image based on this version. Prior to upgrading a particular Jenkins controller whose error message is shared, we ensured all plugins were updated. We then modified the Deployment to use the newly built image (2.479.3).

However, upon accessing the Jenkins Dashboard post-upgrade, we encountered the following error. Please note that the **config.xml** file is present in the Jenkins controller. For reference, we have also attached the output of the plugins directory and the metadata file of the SAML plugin from the same Jenkins controller.

We also noticed some Jenkins Masters are upgrading to version 2.479.3 without any issues that too even without updating the plugins.

We are actively investigating the root cause of the issue and would greatly appreciate your expertise :smiling_face:. With over 4,000 Jenkins instances scheduled for upgrade, it is critical for us to proceed with confidence to avoid any potential disruption to our stakeholders. Your support in resolving this would be highly valuable.

Error

com.thoughtworks.xstream.mapper.CannotResolveClassException: org.jenkinsci.plugins.saml.SamlSecurityRealm
	at com.thoughtworks.xstream.mapper.DefaultMapper.realClass(DefaultMapper.java:81)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
	at com.thoughtworks.xstream.mapper.DynamicProxyMapper.realClass(DynamicProxyMapper.java:55)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
	at com.thoughtworks.xstream.mapper.PackageAliasingMapper.realClass(PackageAliasingMapper.java:88)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
	at hudson.util.RobustReflectionConverter.determineType(RobustReflectionConverter.java:521)
	at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:346)
Caused: jenkins.util.xstream.CriticalXStreamException: 
---- Debugging information ----
cause-exception     : com.thoughtworks.xstream.mapper.CannotResolveClassException
cause-message       : org.jenkinsci.plugins.saml.SamlSecurityRealm
class               : hudson.model.Hudson
required-type       : hudson.model.Hudson
converter-type      : hudson.util.RobustReflectionConverter
path                : /hudson/securityRealm
line number         : 155
version             : 2.479.3
-------------------------------
	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:52)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:136)
	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
Caused: java.io.IOException: Unable to read /var/jenkins_home/config.xml
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused: org.jvnet.hudson.reactor.ReactorException

Caused: hudson.util.HudsonFailedToLoad
	at hudson.WebAppMain$3.run(WebAppMain.java:277)

Output from Plugins directory

ls -l |grep -i saml
drwxr-xr-x 5 jenkins jenkins     4096 Apr  9 10:23 saml
-rw-r--r-- 1 jenkins jenkins  6203145 Aug 18  2023 saml.bak
-rw-r--r-- 1 jenkins jenkins  6036851 Mar 25 05:36 saml.jpi
-rw-r--r-- 1 jenkins jenkins        0 Apr  9 10:23 saml.jpi.pinned
-rw-r--r-- 1 jenkins jenkins       21 Apr  9 10:23 saml.jpi.version_from_image

cat saml.jpi.version_from_image
4.525.v4f6a_7209447e

SAML Manifest file from the Jenkins controller which throw error on upgrading to 2.479.3

Manifest-Version: 1.0
Created-By: Maven Archiver 3.6.0
Build-Jdk-Spec: 17
Specification-Title: SAML Plugin
Specification-Version: 0.0
Implementation-Title: SAML Plugin
Implementation-Version: 4.525.v4f6a_7209447e
Group-Id: org.jenkins-ci.plugins
Artifact-Id: saml
Short-Name: saml
Long-Name: SAML Plugin
Url: https://github.com/jenkinsci/saml-plugin
Compatible-Since-Version: 3.343.vb_63a_6c3df23c
Plugin-Version: 4.525.v4f6a_7209447e
Hudson-Version: 2.479.1
Jenkins-Version: 2.479.1
Plugin-Dependencies: apache-httpcomponents-client-5-api:5.4-136.v5a_2177
 9c63f8,commons-lang3-api:3.17.0-84.vb_b_938040b_078,commons-text-api:1.
 13.0-153.v91dcd89e2a_22,jaxb:2.3.9-1,joda-time-api:2.13.0-93.v9934da_29
 b_a_e9,bouncycastle-api:2.30.1.80-256.vf98926042a_9b_,jackson2-api:2.17
 .0-379.v02de8ec9f64c,mailer:489.vd4b_25144138f
Plugin-Developers: Ben McCann:benmccann:,Ivan Fernandez Calvo:ifernandez
 calvo:
Plugin-License-Name: Apache 2.0 License
Plugin-License-Url: https://www.apache.org/licenses/LICENSE-2.0
Plugin-ScmConnection: scm:git:https://github.com/jenkinsci/saml-plugin.g
 it
Plugin-ScmTag: 4f6a7209447eab49a536729a3a74e339197cdba3
Plugin-ScmUrl: https://github.com/jenkinsci/saml-plugin
Implementation-Build: 4f6a7209447eab49a536729a3a74e339197cdba3

Jenkins setup:

Jenkins: 2.462.3
OS: Linux - 6.1.123+
Java: 17.0.12 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
Parameterized-Remote-Trigger:3.2.1
active-directory:2.38
analysis-model-api:12.9.1
ansicolor:1.0.5
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
apache-httpcomponents-client-5-api:5.4-135.v4da_349961256
asm-api:9.7.1-97.v4cc844130d97
authentication-tokens:1.119.v50285141b_7e1
badge:2.3
basic-branch-build-strategies:81.v05e333931c7d
blackduck-detect:10.0.0
blueocean:1.27.16
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.16
blueocean-commons:1.27.16
blueocean-config:1.27.16
blueocean-core-js:1.27.16
blueocean-dashboard:1.27.16
blueocean-display-url:2.4.3
blueocean-events:1.27.16
blueocean-git-pipeline:1.27.16
blueocean-github-pipeline:1.27.16
blueocean-i18n:1.27.16
blueocean-jwt:1.27.16
blueocean-personalization:1.27.16
blueocean-pipeline-api-impl:1.27.16
blueocean-pipeline-editor:1.27.16
blueocean-pipeline-scm-api:1.27.16
blueocean-rest:1.27.16
blueocean-rest-impl:1.27.16
blueocean-web:1.27.16
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.78.1-248.ve27176eb_46cb_
branch-api:2.1206.vd9f35001c95c
caffeine-api:3.1.8-133.v17b_1ff2e0599
checkmarx:2024.3.2
checks-api:2.2.1
cloudbees-bitbucket-branch-source:912.v3b_f74026941c
cloudbees-folder:6.955.v81e2a_35c08d3
cobertura:1.17
code-coverage-api:4.99.0
command-launcher:116.vd85919c54a_d6
commons-compress-api:1.26.1-2
commons-lang3-api:3.17.0-84.vb_b_938040b_078
commons-text-api:1.13.0-150.vfc0d7966fc38
config-file-provider:980.v88956a_a_5d6a_d
configuration-as-code:1873.vea_5814ca_9c93
copyartifact:761.vea_2b_25523e84
coverage:1.16.1
credentials:1384.vf0a_2ed06f9c6
credentials-binding:681.vf91669a_32e45
customizable-header:169.v671208540e56
dashboard-view:2.523.v673d549dfee5
data-tables-api:2.1.8-1
dependency-check-jenkins-plugin:5.6.0
discard-old-build:1.07
disk-usage:1.3
display-url-api:2.204.vf6fddd8a_8b_e9
docker-commons:445.v6b_646c962a_94
docker-workflow:580.vc0c340686b_54
durable-task:581.v299a_5609d767
echarts-api:5.5.1-4
eddsa-api:0.3.0-4.v84c6f0f4969e
email-ext:1849.v6dd03b_f6e423
embeddable-build-status:487.va_0ef04c898a_2
extended-read-permission:61.vf24570ff3b_e9
favorite:2.221.v19ca_666b_62f5
flatpickr-api:4.6.13-5.v534d8025a_a_59
font-awesome-api:6.6.0-2
forensics-api:2.7.0
git:5.5.2
git-client:5.0.2
git-server:126.v0d945d8d2b_39
github:1.40.0
github-api:1.321-468.v6a_9f5f2d5a_7e
github-branch-source:1807.v50351eb_7dd13
github-oauth:597.ve0c3480fcb_d0
global-slack-notifier:1.5
groovy:457.v99900cb_85593
groovy-postbuild:267.va_df06de9fa_fa_
gson-api:2.11.0-85.v1f4e87273c33
h2-api:11.1.4.199-30.v1c64e772f3a_c
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
htmlpublisher:1.37
http_request:1.19
influxdb:4.0
instance-identity:201.vd2a_b_5a_468a_a_6
ionicons-api:74.v93d5eb_813d5f
jackson2-api:2.17.0-379.v02de8ec9f64c
jacoco:3.3.7
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javax-activation-api:1.2.0-7
javax-mail-api:1.6.2-10
jaxb:2.3.9-1
jdk-tool:80.v8a_dee33ed6f0
jenkins-design-language:1.27.16
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
job-dsl:1.89
joda-time-api:2.13.0-93.v9934da_29b_a_e9
jquery3-api:3.7.1-2
jsch:0.2.16-86.v42e010d9484b_
json-api:20241224-119.va_dca_a_b_ea_7da_5
json-path-api:2.9.0-118.v7f23ed82a_8b_8
junit:1312.v1a_235a_b_94a_31
kubernetes:4306.vc91e951ea_eb_d
kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
kubernetes-credentials:190.v03c305394deb_
lockable-resources:1327.ved786b_a_197e0
mailer:488.v0c9639c1a_eb_3
mask-passwords:173.v6a_077a_291eb_5
matrix-auth:3.2.3
matrix-project:839.vff91cd7e3a_b_2
metrics:4.2.21-458.vcf496cb_839e4
mina-sshd-api-common:2.14.0-138.v6341ee58e1df
mina-sshd-api-core:2.14.0-138.v6341ee58e1df
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
performance:962.v95a_4913d332e
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-github:2.8-159.09e4403bc62f
pipeline-githubnotify-step:49.vf37bf92d2bc8
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-groovy-lib:749.v70084559234a_
pipeline-input-step:508.v584c0e9a_2177
pipeline-maven:1469.ve15ca_a_b_90b_44
pipeline-maven-api:1469.ve15ca_a_b_90b_44
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2218.v56d0cda_37c72
pipeline-model-definition:2.2218.v56d0cda_37c72
pipeline-model-extensions:2.2218.v56d0cda_37c72
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2218.v56d0cda_37c72
pipeline-stage-view:2.34
pipeline-utility-steps:2.18.0
plain-credentials:183.va_de8f1dd5a_2b_
plugin-util-api:5.1.0
prism-api:1.29.0-18
pubsub-light:1.18
purge-build-queue-plugin:88.v23b_97b_f2c7a_d
purge-job-history:74.vf21030329dda_
rebuild:332.va_1ee476d8f6d
resource-disposer:0.24
robot:5.0.0
saferestart:101.vc7fa_8ca_dd18b_
saml:4.464.vea_cb_75d7f5e0
scm-api:696.v778d637b_a_762
script-security:1369.v9b_98a_4e95b_2d
sidebar-link:2.4.1
slack:751.v2e44153c8fe1
snakeyaml-api:2.3-123.v13484c65210a_
sonar:2.17.3
sse-gateway:1.27
ssh-agent:376.v8933585c69d3
ssh-credentials:349.vb_8b_6b_9709f5b_
ssh-slaves:2.973.v0fa_8c0dea_f9f
sshd:3.330.vc866a_8389b_58
structs:338.v848422169819
timestamper:1.27
token-macro:400.v35420b_922dcb_
trilead-api:2.147.vb_73cc728a_32e
variant:60.v7290fc0eb_b_cd
warnings-ng:11.12.0
webhook-step:342.v620877effe14
workflow-aggregator:600.vb_57cdd26fdd7
workflow-api:1358.vfb_5780da_64cb_
workflow-basic-steps:1058.vcb_fc1e3a_21a_9
workflow-cps:4009.v0089238351a_9
workflow-durable-task-step:1378.v6a_3e903058a_3
workflow-job:1436.vfa_244484591f
workflow-multibranch:795.ve0cb_1f45ca_9a_
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:678.v3ee58b_469476
workflow-support:943.v8b_0d01a_7b_a_08
ws-cleanup:0.47

I’ll ping you internally

1 Like