Jenkins upgrade issue from 2.479 to 2.501 - problem with scm-api

[daipok]

Hi all,

We are planning for an upgrade from 2.479 to 2.501, but facing a problem during the jenkins start up.

We use docker to run our jenkins.

Basically when the docker starts up we see the error:

2025-04-02 19:06:05.931	
 - Update required: SCM API Plugin (scm-api 698.v8e3b_c788f0a_6) to be updated to 703.v72ff4b_259600 or higher
2025-04-02 19:06:05.931	
java.io.IOException: Failed to load: Pipeline: Groovy (workflow-cps 4043.va_fb_de6a_a_8b_f5)

And many plugins depend on this Groovy plugins so the upgrade failed basically. We check the image that we deploy, definitely has the latest scm api (SCM API at v704) so we don’t know whether it’s an “sequence issue” that it’s not installed before the above log showing up.

We tried to reproduce this issue locally but we were not able to reproduce. Just try to get some idea for whether there are issues like:

1. Is there a sequence issue, or ways that we can force the scm-api to be updated first?
2. Anyone having this kind of similar issues before? Anything about caching issues?
3. We are trying to reproduce the same symptom in our local environment but we failed to do so.

Jenkins setup:
Jenkins: 2.479
OS: Linux - 4.18.0-553.34.1.el8_10.x86_64
Java: 17.0.12 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
ace-editor:1.1
active-directory:2.38
allure-jenkins-plugin:2.32.0
analysis-model-api:12.9.1
ansible:403.v8d0ca_dcb_b_502
ansicolor:1.0.5
ant:511.v0a_a_1a_334f41b_
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
apache-httpcomponents-client-5-api:5.4-136.v5a_21779c63f8
artifactory:4.0.0
asm-api:9.7.1-97.v4cc844130d97
audit-trail:382.vf64d6f626060
authentication-tokens:1.119.v50285141b_7e1
authorize-project:1.8.1
autocancel:1.0.5
aws-java-sdk:1.12.772-477.v650d756dcf6d
aws-java-sdk-api-gateway:1.12.772-477.v650d756dcf6d
aws-java-sdk-autoscaling:1.12.772-477.v650d756dcf6d
aws-java-sdk-cloudformation:1.12.772-477.v650d756dcf6d
aws-java-sdk-cloudfront:1.12.772-477.v650d756dcf6d
aws-java-sdk-cloudwatch:1.12.772-477.v650d756dcf6d
aws-java-sdk-codebuild:1.12.772-477.v650d756dcf6d
aws-java-sdk-codedeploy:1.12.772-477.v650d756dcf6d
aws-java-sdk-ec2:1.12.772-477.v650d756dcf6d
aws-java-sdk-ecr:1.12.772-477.v650d756dcf6d
aws-java-sdk-ecs:1.12.772-477.v650d756dcf6d
aws-java-sdk-efs:1.12.772-477.v650d756dcf6d
aws-java-sdk-elasticbeanstalk:1.12.772-477.v650d756dcf6d
aws-java-sdk-elasticloadbalancingv2:1.12.772-477.v650d756dcf6d
aws-java-sdk-iam:1.12.772-477.v650d756dcf6d
aws-java-sdk-kinesis:1.12.772-477.v650d756dcf6d
aws-java-sdk-lambda:1.12.772-477.v650d756dcf6d
aws-java-sdk-logs:1.12.772-477.v650d756dcf6d
aws-java-sdk-minimal:1.12.772-477.v650d756dcf6d
aws-java-sdk-organizations:1.12.772-477.v650d756dcf6d
aws-java-sdk-secretsmanager:1.12.772-477.v650d756dcf6d
aws-java-sdk-sns:1.12.772-477.v650d756dcf6d
aws-java-sdk-sqs:1.12.772-477.v650d756dcf6d
aws-java-sdk-ssm:1.12.772-477.v650d756dcf6d
badge:2.3
basic-branch-build-strategies:190.v343a_ee70d920
bitbucket:254.v9d8ec33a_7a_42
blueocean:1.27.16
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.16
blueocean-commons:1.27.16
blueocean-config:1.27.16
blueocean-core-js:1.27.16
blueocean-dashboard:1.27.16
blueocean-display-url:2.4.3
blueocean-events:1.27.16
blueocean-git-pipeline:1.27.16
blueocean-github-pipeline:1.27.16
blueocean-i18n:1.27.16
blueocean-jwt:1.27.16
blueocean-personalization:1.27.16
blueocean-pipeline-api-impl:1.27.16
blueocean-pipeline-editor:1.27.16
blueocean-pipeline-scm-api:1.27.16
blueocean-rest:1.27.16
blueocean-rest-impl:1.27.16
blueocean-web:1.27.16
bootstrap4-api:4.6.0-6
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.78.1-248.ve27176eb_46cb_
branch-api:2.1206.vd9f35001c95c
build-blocker-plugin:166.vc82fc20b_a_ed6
build-monitor-plugin:1.14-948.va_a_a_5a_7eb_1f37
build-name-setter:2.4.3
build-timeout:1.33
build-timestamp:1.0.4
build-token-root:151.va_e52fe3215fc
build-user-vars-plugin:182.v378b_9f14b_487
build-with-parameters:76.v9382db_f78962
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.2.2
claim:554.va_f9b_58b_0a_088
cloudbees-bitbucket-branch-source:912.v3b_f74026941c
cloudbees-folder:6.955.v81e2a_35c08d3
cobertura:1.17
code-coverage-api:4.99.0
command-launcher:116.vd85919c54a_d6
commons-compress-api:1.26.1-2
commons-lang3-api:3.17.0-84.vb_b_938040b_078
commons-text-api:1.13.0-150.vfc0d7966fc38
concurrent-step:1.0.0
config-file-provider:980.v88956a_a_5d6a_d
configuration-as-code:1873.vea_5814ca_9c93
confluence-publisher:163.vf906edb_73cce
copyartifact:761.vea_2b_25523e84
coverage:1.16.1
credentials:1389.vd7a_b_f5fa_50a_2
credentials-binding:687.v619cb_15e923f
cucumber-reports:5.8.5
data-tables-api:2.1.8-1
dependency-check-jenkins-plugin:5.6.0
display-url-api:2.209.v582ed814ff2f
docker-commons:445.v6b_646c962a_94
docker-workflow:592.v1001d948426c
dtkit-api:3.0.3
durable-task:581.v299a_5609d767
echarts-api:5.5.1-4
eddsa-api:0.3.0-4.v84c6f0f4969e
email-ext:1849.v6dd03b_f6e423
embeddable-build-status:487.va_0ef04c898a_2
external-monitor-job:215.v2e88e894db_f8
favorite:2.221.v19ca_666b_62f5
file-parameters:339.v4b_cc83e11455
font-awesome-api:6.6.0-2
forensics-api:2.7.0
gatling:1.3.0
git:5.6.0
git-client:6.1.0
git-parameter:0.10.0
git-server:126.v0d945d8d2b_39
github:1.40.0
github-api:1.321-478.vc9ce627ce001
github-autostatus:3.6.2
github-branch-source:1807.v50351eb_7dd13
golang:1.4
gradle:2.14
groovy:457.v99900cb_85593
gson-api:2.11.0-85.v1f4e87273c33
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
htmlpublisher:1.37
http_request:1.19
ignore-committer-strategy:29.v7c3891a_434c3
influxdb:3.4
instance-identity:201.vd2a_b_5a_468a_a_6
ionicons-api:74.v93d5eb_813d5f
ivy:2.8
jackson2-api:2.17.0-379.v02de8ec9f64c
jacoco:3.3.7
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javadoc:280.v050b_5c849f69
javax-activation-api:1.2.0-7
javax-mail-api:1.6.2-10
jaxb:2.3.9-1
jdk-tool:80.v8a_dee33ed6f0
jenkins-design-language:1.27.16
jersey2-api:2.44-151.v6df377fff741
jfrog:1.5.6
jira:3.13
jira-steps:2.0.165.v8846cf59f3db
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
jnr-posix-api:3.1.20-125.vb_6ec4b_21b_15e
job-dsl:1.89
jobConfigHistory:1294.v961a_b_707546a_
joda-time-api:2.13.0-93.v9934da_29b_a_e9
jquery:1.12.4-3
jquery3-api:3.7.1-2
jsch:0.2.16-86.v42e010d9484b_
json-api:20241224-119.va_dca_a_b_ea_7da_5
json-path-api:2.9.0-118.v7f23ed82a_8b_8
junit:1312.v1a_235a_b_94a_31
kubernetes:4306.vc91e951ea_eb_d
kubernetes-cli:1.12.1
kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
kubernetes-credentials:190.v03c305394deb_
ldap:759.vef7f616475df
leastload:62.vfa_8830902733
lockable-resources:1327.ved786b_a_197e0
mailer:489.vd4b_25144138f
mask-passwords:173.v6a_077a_291eb_5
matrix-auth:3.2.3
matrix-project:840.v812f627cb_578
maven-plugin:3.24
mercurial:1308.v0e0888fdb_628
metrics:4.2.21-458.vcf496cb_839e4
mina-sshd-api-common:2.14.0-143.v2b_362fc39576
mina-sshd-api-core:2.14.0-143.v2b_362fc39576
momentjs:1.1.1
monitoring:1.99.0
msbuild:1.35
mstest:1.0.0
mstestrunner:1.5.0
next-build-number:66.v4b_4762172d53
nunit:563.va_e639b_4b_fc20
nvm-wrapper:0.1.7
okhttp-api:4.11.0-181.v1de5b_83857df
openshift-client:1.0.37
pam-auth:1.11
parameterized-scheduler:277.v61a_4b_a_49a_c5c
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-groovy-lib:749.v70084559234a_
pipeline-input-step:508.v584c0e9a_2177
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2218.v56d0cda_37c72
pipeline-model-definition:2.2218.v56d0cda_37c72
pipeline-model-extensions:2.2218.v56d0cda_37c72
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2218.v56d0cda_37c72
pipeline-stage-view:2.34
pipeline-utility-steps:2.18.0
plain-credentials:183.va_de8f1dd5a_2b_
plugin-util-api:5.1.0
popper-api:1.16.1-3
popper2-api:2.11.6-5
powershell:2.2
prism-api:1.29.0-18
prometheus:811.v1823c780b_e8e
pubsub-light:1.18
pyenv-pipeline:2.1.2
rebuild:332.va_1ee476d8f6d
remote-file:1.24
resource-disposer:0.24
role-strategy:743.v142ea_b_d5f1d3
run-condition:1.7
scm-api:698.v8e3b_c788f0a_6
scm-filter-branch-pr:210.vf35256163d30
script-security:1369.v9b_98a_4e95b_2d
simple-theme-plugin:196.v96d9592f4efa_
sloccount:1.27
snakeyaml-api:2.3-123.v13484c65210a_
sonar:2.18
splunk-devops:1.10.1
splunk-devops-extend:1.10.1
sse-gateway:1.27
ssh-credentials:349.vb_8b_6b_9709f5b_
ssh-steps:2.0.68.va_d21a_12a_6476
sshd:3.330.vc866a_8389b_58
stashNotifier:1.516.v6d49910f22a_5
structs:338.v848422169819
test-results-analyzer:0.4.0
timestamper:1.27
token-macro:400.v35420b_922dcb_
trilead-api:2.147.vb_73cc728a_32e
uno-choice:2.8.4
variant:60.v7290fc0eb_b_cd
violation-comments-to-stash:1.134
vstestrunner:1.0.8
warnings-ng:11.12.0
windows-slaves:1.8.1
workflow-aggregator:600.vb_57cdd26fdd7
workflow-api:1358.vfb_5780da_64cb_
workflow-basic-steps:1058.vcb_fc1e3a_21a_9
workflow-cps:4009.v0089238351a_9
workflow-cps-global-lib:612.v55f2f80781ef
workflow-durable-task-step:1378.v6a_3e903058a_3
workflow-job:1447.v559b_c710cd2e
workflow-multibranch:795.ve0cb_1f45ca_9a_
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:678.v3ee58b_469476
workflow-support:943.v8b_0d01a_7b_a_08
ws-cleanup:0.47
xunit:3.1.5

[mawinter69 (Markus Winter)]

No this is not a sequence issue.
You should double check if your image is really correct. Do you have a fixed docker image, means in the image are there already all plugins contained or are you using a docker image that during startup downloads the plugins via the plugin-installation-manager tool.

Besides that, you should do some housekeeping and remove obsolete plugins, e.g. ace-editor and handlebars, popper2-api, popper-api, bootstrap4-api. Also you have active-directory, ldap and pam-auth installed, you need only one of them. Also role-strategy and matrix-auth, you only need one of them. There’s probably more plugins you can remove. But removing them will not solve the issue you have.

[daipok]

Yes there are quite a few obsoleted plugins that get to be cleaned but I don’t think they are the cause to my issue.

The image I have already have all the plugins contained, so they are not downloading plugins during start up (I can see them all /usr/share/jenkins/ref/plugins)

This approach we use (docker, plugins…etc) had been going fine for numerous upgrades but this time for some reason having this hiccup.

[MarkEWaite (Mark Waite)]

I suspect that something is wrong with your Dockerfile definition or with how you are performing the update. I was able to create a 2.501 Jenkins controller with your set of plugins (including 211 that need to be updated and 3 that have security vulnerabilities and 12 that are deprecated) with the following Dockerfile:

FROM jenkins/jenkins:2.501
RUN echo 2.0 > /usr/share/jenkins/ref/jenkins.install.UpgradeWizard.state
COPY --chown=jenkins:jenkins plugins.txt /usr/share/jenkins/ref/plugins.txt
RUN jenkins-plugin-cli -f /usr/share/jenkins/ref/plugins.txt

[daipok]

The most puzzling thing is we try the image locally, and working as well. How we did it is to deploy an image with the existing set of plugins, and then upgrade to a new image with the new set of plugins, and all working fine.

However, when we upgrade on the live environment then we got the weird error that the scm-api dependency fails (while the scm-api latest 704 is in the list of plugins).

[MarkEWaite (Mark Waite)]

@daipok

However, when we upgrade on the live environment then we got the weird error that the scm-api dependency fails (while the scm-api latest 704 is in the list of plugins).

I think that means that your live environment is still running the older scm-api plugin and is not replacing the older version with a newer version during the upgrade.

[daipok]

yes the current live env is running 698, and the intention is to upgrade it to 704 with the new image.

However, it just somehow fails to upgrade, and the groovy plugin is already complaining.