Upgrading jenkins to LTS version 2.452.2

[snehalhingane]

Hi All,
We are upgrading our jenkins from 2.332.3 to 2.452.2. we are able to successfully upgrade it to 2.452.2, everything works perfectly fine. only issue in “Enable project-based security” option from job config where it is not allowing to add users for giving release permission

Observation:

• snakeyaml-api plugins auto updated to lts version i.e. 2.2-111.vc6598e30cc65 which is expected based on the changeslog: LTS Changelog
• We already have snakeyaml-api plugin v1.30.1 in plugins.txt, one observation is, when we do first deployment it updates snakeyml-api to latest version as per base image and we are able to add the users in “Enable project-based security” inside job config, but issue here is when we build any job it fails with below issue
  13:13:41 Also: org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: eaa613f0-6af2-4a41-a952-5c37ce405b7513:13:41 java.lang.NoSuchMethodError: org.yaml.snakeyaml.constructor.SafeConstructor: method 'void <init>()' not found
• this issue is already reported in latest version i.e. 2.2-111.vc6598e30cc65 & for fixing this we need to downgrade it to make it work.
• Official document has instruction to not to use latest version as it has breaking changes: SnakeYAML API
  Attaching list of plugins & error logs
  Logs:

2024-06-24 10:26:03.951+0000 [id=304]	INFO	c.a.c.util.logging.ClientLogger#performLogging: Azure Identity => getToken() result for scopes [https://graph.microsoft.com/.default]: SUCCESS
2024-06-24 10:26:04.108+0000 [id=21]	SEVERE	c.m.graph.logger.DefaultLogger#logError: CoreHttpProvider[sendRequestInternal] - 396Graph service exception Error code: Request_ResourceNotFound
2024-06-24 10:26:04.109+0000 [id=21]	SEVERE	c.m.graph.logger.DefaultLogger#logError: CoreHttpProvider[sendRequestInternal] - 396Error message: Resource 'ABC-MULTIBRANCH' does not exist or one of its queried reference-property objects are not present.
2024-06-24 10:26:04.110+0000 [id=21]	SEVERE	c.m.graph.logger.DefaultLogger#logError: CoreHttpProvider[sendRequestInternal] - 396
2024-06-24 10:26:04.110+0000 [id=21]	SEVERE	c.m.graph.logger.DefaultLogger#logError: CoreHttpProvider[sendRequestInternal] - 396GET https://graph.microsoft.com/v1.0/users/TOF-MULTIBRANCH
2024-06-24 10:26:04.110+0000 [id=21]	SEVERE	c.m.graph.logger.DefaultLogger#logError: CoreHttpProvider[sendRequestInternal] - 396SdkVersion : graph-java/v3.8.0
2024-06-24 10:26:04.111+0000 [id=21]	SEVERE	c.m.graph.logger.DefaultLogger#logError: CoreHttpProvider[sendRequestInternal] - 396
2024-06-24 10:26:04.111+0000 [id=21]	SEVERE	c.m.graph.logger.DefaultLogger#logError: CoreHttpProvider[sendRequestInternal] - 396
2024-06-24 10:26:04.111+0000 [id=21]	SEVERE	c.m.graph.logger.DefaultLogger#logError: CoreHttpProvider[sendRequestInternal] - 396404 : Not Found
2024-06-24 10:26:04.112+0000 [id=21]	SEVERE	c.m.graph.logger.DefaultLogger#logError: CoreHttpProvider[sendRequestInternal] - 396[...]
2024-06-24 10:26:04.112+0000 [id=21]	SEVERE	c.m.graph.logger.DefaultLogger#logError: CoreHttpProvider[sendRequestInternal] - 396
2024-06-24 10:26:04.112+0000 [id=21]	SEVERE	c.m.graph.logger.DefaultLogger#logError: CoreHttpProvider[sendRequestInternal] - 396[Some information was truncated for brevity, enable debug logging for more details]
2024-06-24 10:26:04.113+0000 [id=21]	SEVERE	c.m.graph.logger.DefaultLogger#logError: Throwable detail: com.microsoft.graph.http.GraphServiceException: Error code: Request_ResourceNotFound
Error message: Resource 'TOF-MULTIBRANCH' does not exist or one of its queried reference-property objects are not present.

GET https://graph.microsoft.com/v1.0/users/TOF-MULTIBRANCH
SdkVersion : graph-java/v3.8.0
404 : Not Found

List of plugins which we are using as part of plugin.txt

ace-editor:1.1
active-directory:2.25.1
analysis-core:1.96
analysis-model-api:10.10.1
ansible:1.1
ant:475.vf34069fef73c
antisamy-markup-formatter:2.7
AnchorChain:1.0
apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5
audit-trail:3.10
authentication-tokens:1.4
avatar:1.2
aws-credentials:191.vcb_f183ce58b_9
aws-java-sdk-cloudformation:1.12.215-339.vdc07efc5320c
aws-java-sdk-codebuild:1.12.215-339.vdc07efc5320c
aws-java-sdk-ec2:1.12.215-339.vdc07efc5320c
aws-java-sdk-ecr:1.12.215-339.vdc07efc5320c
aws-java-sdk-ecs:1.12.215-339.vdc07efc5320c
aws-java-sdk-elasticbeanstalk:1.12.215-339.vdc07efc5320c
aws-java-sdk-iam:1.12.215-339.vdc07efc5320c
aws-java-sdk-logs:1.12.215-339.vdc07efc5320c
aws-java-sdk-minimal:1.12.215-339.vdc07efc5320c
aws-java-sdk-ssm:1.12.215-339.vdc07efc5320c
aws-java-sdk:1.12.215-339.vdc07efc5320c
azure-ad:185.v3b416408dcb1
azure-sdk:106.v552de1e64d56
badge:1.9.1
batch-task:1.19
blame-upstream-commiters:1.2
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.25.5
blueocean-commons:1.25.5
blueocean-config:1.25.5
blueocean-core-js:1.25.5
blueocean-dashboard:1.25.5
blueocean-display-url:2.4.1
blueocean-events:1.25.5
blueocean-executor-info:1.25.5
blueocean-git-pipeline:1.25.5
blueocean-github-pipeline:1.25.5
blueocean-i18n:1.25.5
blueocean-jira:1.25.5
blueocean-jwt:1.25.5
blueocean-personalization:1.25.5
blueocean-pipeline-api-impl:1.25.5
blueocean-pipeline-editor:1.25.5
blueocean-pipeline-scm-api:1.25.5
blueocean-rest-impl:1.25.5
blueocean-rest:1.25.5
blueocean-web:1.25.5
blueocean:1.25.5
bootstrap4-api:4.6.0-5
bootstrap5-api:5.2.1-3
bouncycastle-api:2.30.1.78.1-233.vfdcdeb_0a_08a_a_
branch-api:2.1046.v0ca_37783ecc5
build-failure-analyzer:2.3.0
build-flow-plugin:0.19
build-keeper-plugin:1.3
build-monitor-plugin:1.13+build.202205140447
build-name-setter:2.2.0
build-pipeline-plugin:1.5.8
build-timeout:1.20
build-timestamp:1.0.3
build-user-vars-plugin:1.8
built-on-column:1.1
caffeine-api:2.9.3-65.v6a_47d0f4d1fe
categorized-view:1.12
changelog-history:1.7
checkmarx:2022.4.3
checks-api:1.8.1
checkstyle:4.0.0
ci-game:1.26
claim:2.18.2
cloudbees-bitbucket-branch-source:773.v4b_9b_005b_562b_
cloudbees-folder:6.815.v0dd5a_cb_40e0e
codecover:1.1
command-launcher:81.v9c2cb_cb_db_392
compress-artifacts:1.10
compress-buildlog:1.2
conditional-buildstep:1.4.2
config-file-provider:3.10.0
configurationslicing:430.v966357576543
convert-to-pipeline:1.0
copy-data-to-workspace-plugin:1.0
copy-project-link:58.v13dd4905b_c17
copy-to-agent:1.4.4
copyartifact:1.46.4
credentials-binding:604.vb_64480b_c56ca_
credentials:1307.v3757c78f17c3
cucumber-reports:5.7.0
customize-build-now:1.1
cvs:2.19.1
dashboard-view:2.432.va_712ce35862d
data-tables-api:1.11.4-4
dependencyanalyzer:0.7
depgraph-view:1.0.5
description-setter:1.10
display-url-api:2.3.7
docker-build-step:2.8
docker-commons:1.19
docker-java-api:3.1.5.2
docker-plugin:1.2.9
docker-slaves:1.0.7
docker-workflow:1.28
downstream-buildview:1.9
downstream-ext:1.8
dropdown-viewstabbar-plugin:1.7
durable-task:496.va67c6f9eefa7
echarts-api:5.4.0-1
email-ext:2.88
embeddable-build-status:2.0.3
emma:1.31
envinject-api:1.199.v3ce31253ed13
envinject:2.866.v5c0403e3d4df
environment-script:1.2.6
extended-choice-parameter:346.vd87693c5a_86c
extended-read-permission:3.2
extensible-choice-parameter:1.8.0
external-monitor-job:191.v363d0d1efdf8
extra-columns:1.25
fail-the-build-plugin:1.0
favorite:2.4.1
findbugs:5.0.0
font-awesome-api:6.2.0-3
forensics-api:1.13.0
gerrit-code-review:0.4.7
gerrit-trigger:2.36.0
gerrit-verify-status-reporter:0.0.3
git-changelog:3.23
git-client:4.2.0
git-parameter:0.9.16
git-server:1.11
github-api:1.303-400.v35c2d8258028
github-branch-source:1628.vb_2f51293cb_78
github-organization-folder:1.6
github:1.34.3
git:5.0.2
global-build-stats:1.5
google-git-notes-publisher:0.3
google-oauth-plugin:1.0.6
gradle:1.38
greenballs:1.15.1
groovy-postbuild:2.5
groovy:2.4
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
heavy-job:1.1
htmlpublisher:1.30
hudson-pview-plugin:1.8
icon-shim:3.0.0
influxdb:3.2
instant-messaging:1.49
iphoneview:0.2
ivy:2.2
jackson2-api:2.14.2-319.v37853346a_229
jacoco:3.3.2
javadoc:217.v905b_86277a_2a_
javatest-report:1.6
javax-activation-api:1.2.0-3
javax-mail-api:1.6.2-6
jaxb:2.3.6-1
jdk-tool:1.5
jenkins-design-language:1.25.5
jenkins-jira-issue-updater:1.18
jenkins-multijob-plugin:1.36
jira:3.7.1
JiraTestResultReporter:169.v6073bb438046
jjwt-api:0.11.2-71.v2722b_b_06a_2a_f
jnr-posix-api:3.1.7-3
job-import-plugin:3.5
job-poll-action-plugin:1.0
jobConfigHistory:1139.v888b_656ca_f6d
jobcopy-builder:1.4.2
jquery-detached:1.2.1
jquery-ui:1.0.2
jquery3-api:3.6.0-4
jquery:1.12.4-1
jsch:0.1.55.2
junit:1189.v1b_e593637fa_e
kubernetes-client-api:5.12.2-193.v26a_6078f65a_9
kubernetes-credentials:0.9.0
kubernetes:3724.v0920c1e0ec69
ldap:2.10
lockable-resources:2.15
m2release:0.16.3
mailer:448.v5b_97805e3767
mapdb-api:1.0.9.0
mask-passwords:3.1
matrix-auth:3.1.2
matrix-combinations-parameter:1.3.1
matrix-project:789.v57a_725b_63c79
matrixtieparent:1.2
maven-plugin:3.18
mercurial:2.16.2
metrics:4.1.6.2
miniorange-saml-sp:1.0.10
momentjs:1.1.1
monitoring:1.91.0
mttr:1.1
multi-branch-project-plugin:0.7
multiple-scms:0.8
naginator:1.18.1
nested-view:1.17
nexus-jenkins-plugin:3.14.412.v8021dc9cc4ef
no-agent-job-purge:1.2
nodelabelparameter:1.10.3.1
oauth-credentials:0.5
okhttp-api:4.9.3-105.vb96869f8ac3a
Office-365-Connector:4.17.0
pam-auth:1.8
parameterized-trigger:2.44
pipeline-aggregator-view:1.11
pipeline-build-step:2.18
pipeline-github-lib:36.v4c01db_ca_ed16
pipeline-graph-analysis:195.v5812d95a_a_2f9
pipeline-groovy-lib:589.vb_a_b_4a_a_8c443c
pipeline-input-step:448.v37cea_9a_10a_70
pipeline-milestone-step:101.vd572fef9d926
pipeline-model-api:2.2121.vd87fb_6536d1e
pipeline-model-declarative-agent:1.1.1
pipeline-model-definition:2.2081.v3919681ffc1e
pipeline-model-extensions:2.2121.vd87fb_6536d1e
pipeline-rest-api:2.24
pipeline-stage-step:293.v200037eefcd5
pipeline-stage-tags-metadata:2.2081.v3919681ffc1e
pipeline-stage-view:2.24
pipeline-utility-steps:2.12.1
PrioritySorter:4.1.0
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:2.20.0
pmd:4.0.0
pollscm:1.3.1
popper-api:1.16.1-3
popper2-api:2.11.6-1
postbuild-task:1.9
prism-api:1.28.0-2
project-stats-plugin:0.4
publish-over-cifs:0.16
publish-over-ftp:1.17
publish-over-ssh:1.24
publish-over:0.22
pubsub-light:1.18
purge-build-queue-plugin:46.v90db_9fb_8b_f4a_
rebuild:1.34
release:2.14
resource-disposer:0.19
role-strategy:484.v8a_a_e4b_d785fd
run-condition:1.5
scm-api:667.v8b_6e07cdc7f2
scm-sync-configuration:0.0.10
scp:1.8
screenshot:1.1
script-realm:1.5
script-security:1244.ve463715a_f89c
scriptler:3.5
scripttrigger:0.34
sectioned-view:1.25
sitemonitor:0.6
skype-notifier:1.1.0
agent-setup:1.10
agent-status:1.6
snakeyaml-api:1.30.1
sonar:2.14
sse-gateway:1.25
ssh-agent:295.v9ca_a_1c7cc3a_a_
ssh-credentials:337.v395d2403ccd4
ssh-slaves:1.814.vc82988f54b_10
ssh:2.6.1
sshd:3.237.v883d165a_c1d3
started-by-envvar:1.0
structs:324.va_f5d6774f3a_d
subversion:2.15.5
svn-tag:1.18
svn-workspace-cleaner:1.1
swarm:3.32
text-finder:1.19
thinBackup:1.10
throttle-concurrents:2.8
timestamper:1.17
token-macro:359.vb_cde11682e0c
toolenv:1.2
translation:1.16
trilead-api:2.84.v72119de229b_7
unique-id:2.2.1
uno-choice:2.6.1
update-sites-manager:2.0.0
variant:60.v7290fc0eb_b_cd
view-job-filters:2.3
warnings-ng:9.12.0
warnings:5.0.2
windows-slaves:1.8.1
wix:1.12
workflow-aggregator:578.vf9a_f99755f4a_
workflow-api:1208.v0cc7c6e0da_9e
workflow-basic-steps:948.v2c72a_091b_b_68
workflow-cps-global-lib:588.v576c103a_ff86
workflow-cps:2729.2732.vda_e3f07b_5a_f8
workflow-durable-task-step:1190.vc93d7d457042
workflow-job:1289.vd1c337fd5354
workflow-multibranch:712.vc169a_1387405
workflow-scm-step:408.v7d5b_135a_b_d49
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:839.v35e2736cfd5c
ws-cleanup:0.42
xfpanel:2.0.1
xvnc:1.28
zentimestamp:4.2

Could you please suggest on the same as its blocking our upgrade

[mawinter69 (Markus Winter)]

I would recommend to update all plugins to the latest version when you update to latest LTS of Jenkins. Especially the new snakeyaml might require to update plugins so they make use of the changed apis of snakeyaml

[snehalhingane]

Hi @mawinter69,
Appreciate for quick response, what we decided was, we first upgrade Jenkins to LTS version and then will upgrade plugins one by one based on the security vulnerabilities, because we have lot of plugins in the plugin.txt which are dependent on each other.
Also everything is working as expected except this release permission
Also I can see, snakeyaml-api doesn’t have any dependency(this is what we see in plugins wiki page) hence we tried to bump up snakeyaml-api plugin from 1.30.1 to latest step by step but non of them is working

Would you still recommend to upgrade all plugins which we have in plugins.txt ??
much appreciated,
Snehal

[mawinter69 (Markus Winter)]

snakeyaml is used by 15 plugins, most important it is used by jackson2-api plugin which is used by many plugins.

So yes I strongly recommend to update all plugins in one go.

And I also recommend to do some cleanup and remove deprecated plugins and plugins that you don’t use
e.g. ace-editor, handlebars, jquery-iu are deprecated and can probably be uninstalled.

warnings,checkstyle and pmd plugins are no longer distributed you already have warnings-ng, you should check if you might have jobs that use warnings or pmd plugin and migrate them to use warnings-ng

windows-slaves is deprecated, check if you have agents that are managed with it and if yes try to use e.g. ssh-slaves plugin instead

greenballs has no effect anymore since a while and can be removed.

you have azure-ad, active-directory, ldap, miniorange-saml-sp and pam-auth installed you need only one of them

You have matrix-auth and role-strategy installed, you can uninstall role-strategy as you don’t use it.

There’s probably more plugins that can be removed because you don’t use them, e.g. you have cvs, svn and git, but my guess is you only use git

[snehalhingane]

Thanks @mawinter69, will try and get back to you If I face any issue

[sodul (Stephane Odul)]

The way I perform the upgrade of our jenkins instances (we upgrade on every major LTS to avoid major versions jumps).

Go to our sandbox instance:

• go to the plugin page and update all the plugins to their known compatible versions of the old LTS. The plugin UI is doing a good job of this nowadays.
• upgrade to the latest LTS.
• go back to the plugins page and upgrade the plugins again, usually we will find a few more plugins that can be upgraded further.
• run smoke test pipelines to ensure the most common things are working as expected.

We have been using this procedure for a while and this seems to work well.

In our case we run all our Jenkins instances under Kubernetes which makes upgrading the other instances a breeze, but once the sandbox instance is working well. There are rarely other unexpected issues we need to address.

[snehalhingane]

Hi @sodul, Yes we are also upgrading jenkins on every major LTS to avoid major versions jumps.
If I understand your steps correctly, so you are saying to upgrade plugins on every major LTS version at a time of upgrading jenkins instead of first upgrading jenkins to latest LTS version and then final step to upgrade plugins, is that correct??

What we have done so far is:

• We have Upgraded Jenkins on every major LTS version
• We have upgraded Jenkins successfully & now we are facing issue with release permission, which I explained in my initial post
• @mawinter69 suggested to upgrade all plugins to latest LTS version

[snehalhingane]

Hi @mawinter69, One thing on which I’m not clear is, you suggested to upgrade all plugins from plugins.txt which includes snakeyaml-api plugin as well, but as I mentioned in earlier post that If we upgrade snakeyaml-api to latest version then its blocking us to run build. The issue is better described at JENKINS-71966.

• They suggested to not to upgrade it to latest version as it has breaking changes: SnakeYAML API
• We will try to upgrade other plugins to latest LTS version but what about snakeyaml-api ???
  Could you please suggest on same??

[mawinter69 (Markus Winter)]

In JENKINS-71966 it is written

The issue can also be avoided by upgrading from Jenkins 2.387.3 to Jenkins 2.401.3 or Jenkins 2.414.1 and upgrading the blue ocean plugins. An updated blue ocean plugin (1.27.6) is available for Jenkins 2.401.3 with support for snakeyaml 2.2

So when you update all plugins I think that everything will work.

[snehalhingane]

Sure, Thanks @mawinter69, will try to upgrade all plugins & see how it goes

[sodul (Stephane Odul)]

Correct, we found out that it leads to more stable upgrades overall. Some older plugins will not work well with newer LTS because of internal plugin API changes, so if you upgrade jenkins.war with older incompatible plugins you have a higher risk of breakage.

• Upgrade plugins to latest compatible versions through the Manage Plugins UI.
• Upgrade jenkins.war.
• Upgrade the plugins the the latest compatible versions.

[snehalhingane]

Hi @mawinter69/@sodul
I upgraded all plugins as per your suggestion & as of now can see 2 issues,

• Can see list of jobs while searching in search bar, but when we click on any of jobs (In our case RELENG-AUTOMATION-MULTIBRANCH), it failed with below issue:
  
  
  
  
  
  
  Solution: This is because of nested view plugin(v1.33), I downgraded to old version 1.7 and it is working, did you face this issue??
• Another issue while running build as below:
  • 16:36:50 ERROR: Failed to launch agent-core-af2f6c91-74b5-49da-98e9-2a1b65956091-ssr5b-4sfxn 16:36:50 sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    
    
    
    can you please suggest on this??

[sodul (Stephane Odul)]

No, we use the Folders plugin, and using the Nested view would not make sense for us.

The Nested View plugin has a low install base (2-3%) nowadays and while it seems to be maintained I would recommend that you consider the newer Folders plugin which has a 95% install base and is more actively maintained.

[snehalhingane]

Hi @sodul / @mawinter69,
Can you please suggest on below issue which we are getting while running builds on jenkins:

• 16:36:50 ERROR: Failed to launch agent-core-af2f6c91-74b5-49da-98e9-2a1b65956091-ssr5b-4sfxn 16:36:50 sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
  
  

This issue occurs when we upgrade all plugins with latest version. Builds are working fine before upgrading plugins to latest version.

• we tried to add cacert using below command inside jre/openjdk/security but it didn’t worked
  $/opt/java/openjdk/bin/keytool -import -alias kubernetes.default -file .crt -keystore cacerts -storepass changeit -noprompt

• We also tried to downgrade kubernetes plugin from 3724.v0920c1e0ec69 to 4253.v7700d91739e5, but when we downgrade it to old version we get authentication error as below
  

  

did you faced this kind of issue, could you please suggest???

[sodul (Stephane Odul)]

We run our Jenkins workload on Kubernetes and are currently on the latest LTS and the k8s plugin is from a few weeks ago. We do not experience this issue.

This are the k8s plugins we run today:

• kubernetes:4246.v5a_12b_1fe120e
• kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
• kubernetes-credentials:174.va_36e093562d9

I’m afraid I won’t be able to help much with regards to SSL errors.

[snehalhingane]

Hi @sodul /@mawinter69
Thanks for sharing your inputs here, new jenkins version has been launch i.e. 2.462.1
We have upgraded to this version now also as per your suggestion updated all plugins to lts version (git version is: 5.2.2)
but we face below isssue, few jobs are working fine but few are failing with below:

Could you please suggest on same??
Thanks in advance

[sodul (Stephane Odul)]

It looks like git init /jenkins/workspace/FEATURE-18-CLEX-26557-CommitStage fails. Something a little odd here is that the directory doing the init is /jenkins/workspace/FEATURE-18-CLEX-26557-CommitStage where FEATURE-18-CLEX-26557-CommitStage should be the same as the job name.

I would get shell access to the /jenkins/workspace/ folder as the same user running the job and run that git init command from the shell.

The stack trace you provided should have included the error output of the git command which might provide more details.

FYI, I do not come to these forums every week, so my replies might be random. I’m just an other user myself.