I am currently in the process of upgrading my Jenkins setup. It is running authentication based on LDAP, with Matrix-based permissions.
I’ll admit I didn’t note down what version Jenkins started out with being before upgrading, nor the plugins ,as I didn’t expect to encounter this problem, but it has currently been upgraded to version 2.516.3 and it had been quite a while since the Jenkins system and plugins had been upgraded. Upgrading from the version I started out with, up and until 2.516.3 didn’t break anything.
However, whenever I try to upgrade to any version above 2.516.3, any attempts at logging in, are subsequently met with the a 403 invalid crumb error
I have read up online about it and as I have understood it, this has to do with some CSRF protection that Jenkins has built in the system. But it also appears to be a very generic issue, which could stem from a multitude of sources.
I am highly certain it is the authentication that breaks, because disabling security by modifying
the line true to false in config.xml and then jumping on the jenkins platform reveals that everything runs fine.
In no particular order, I have tried the following
1. Ensuring all plugins are updated before upgrading Jenkins itself
2. Before upgrading I uninstalled the LDAP plugin, removed any and all old config files in the plugins folder in $JENKINS_HOME/plugins, and the ldap.jpi/ldap.hpi file, reinstalled it and then re-added the users to the system, then upgrading Jenkins.
3. I have read the upgrade guide for Jenkins. As far as I have understood, the next, smallest upgrade step I could take would be upgrading it to 2.528.x None of the upgrading steps are applicable, as per the guide here Upgrading to Jenkins LTS 2.528.x.
4. I have read about the possibility of disabling CSRF protection, but aside from the fact that is seems to be an unwise idea, it doesn’t solve the root of the problem.
5. Trying to log in from a private window doesn’t help either.
6. I have read that if there is a jenkins url mismatch, the crumb will be considered invalid. However, I haven’t found any url values that don’t match up (and it seems odd that an upgrade would change that).
7. I have heard some suggest using API tokens. But that seems only valid for scripts?
8. Someone on Stackoverflow suggest navigating to the logout page, then logging in again. That didn’t work.
9. There are several upgrade guides akin this this one , suggesting you download ldap.hpi and use it to replace the existing ldap.jpi file. I did this as part of doing attempt #2. and that didn’t work.
10. I have also tried first upgrading Jenkins, disabling security, then repeating attempt #2 I.e rebuilding the security setup from scratch after upgrading. That didn’t work either.
11. Finally, as a last resort, I tried to go away entirely from using LDAP as the security realm to using Jenkins’ own user database for managing users. Still, the issue persists.
Other information, that might be useful in troubleshooting:
1. I am using the default crumb issuer, with enabled proxy compatibility.
2. Jenkins is not running in a container, Openshift or anything like that. It is running directly on the system.
3. LDAP definitely works before upgrading. You can add users, and testing LDAP with a valid user by using the “Test LDAP settings” button successfully shows the expected output.
4. Per request of the message that shows when you attempt to make a post, here is the output of the Jenkins script, showing the system’s setup:
Jenkins: 2.516.3
OS: Linux - 5.14.0-611.24.1.el9_7.x86_64
Java: 21.0.9 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)
---
analysis-model-api:13.18.0-935.v784ca_107400a_
ant:520.vd082ecfb_16a_9
antisamy-markup-formatter:173.v680e3a_b_69ff3
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
apache-httpcomponents-client-5-api:5.6-183.ve5a_8a_b_e71e59
asm-api:9.9.1-189.vb_5ef2964da_91
authentication-tokens:1.144.v5ff4a_5ec5c33
blueocean-bitbucket-pipeline:1.27.25
blueocean-commons:1.27.25
blueocean-core-js:1.27.25
blueocean-git-pipeline:1.27.25
blueocean-github-pipeline:1.27.25
blueocean-jwt:1.27.25
blueocean-pipeline-api-impl:1.27.25
blueocean-pipeline-scm-api:1.27.25
blueocean-rest:1.27.25
blueocean-rest-impl:1.27.25
blueocean-web:1.27.25
bootstrap5-api:5.3.8-895.v4d0d8e47fea_d
bouncycastle-api:2.30.1.82-277.v70ca_0b_877184
branch-api:2.1268.v044a_87612da_8
build-notifications:1.5.0
caffeine-api:3.2.3-194.v31a_b_f7a_b_5a_81
checks-api:402.vca_263b_f200e3
cloudbees-bitbucket-branch-source:937.2.3
cloudbees-disk-usage-simple:256.v20ec4eb_884f1
cloudbees-enabler:1.43.v5b_7c0c898e84
cloudbees-folder:6.1053.vd62fb_b_f7367b_
command-launcher:123.v37cfdc92ef67
commons-collections4-api:4.5.0-8.va_d5448ef9011
commons-lang3-api:3.20.0-109.ve43756e2d2b_4
commons-text-api:1.15.0-210.v7480a_da_70b_9e
conditional-buildstep:1.5.0
config-file-provider:1006.vc7366c201f57
credentials:1480.v2246fd131e83
credentials-binding:702.vfe613e537e88
custom-folder-icon:2.21
data-tables-api:2.3.5-1497.v38449eb_7d5a_1
diagnostics:1.2
display-url-api:2.217.va_6b_de84cc74b_
docker-build-publish:1.4.0
docker-commons:457.v0f62a_94f11a_3
durable-task:651.v1f5e074fc83f
echarts-api:6.0.0-1165.vd1283a_3e37d4
eddsa-api:0.3.0.1-29.v67e9a_1c969b_b_
email-ext:1933.v45cec755423f
emoji-symbols-api:17.0-57.v8d44b_9a_b_d5ea_
favorite:2.263.v941d21defef7
font-awesome-api:7.1.0-882.v1dfb_771e3278
forensics-api:3.1832.va_1179842528b_
git:5.9.0
git-client:6.5.0
git-server:137.ve0060b_432302
github:1.45.0
github-api:1.330-492.v3941a_032db_2a_
github-branch-source:1917.v9ee8a_39b_3d0d
github-issues:1.2.4
github-pullrequest:0.7.3
groovy:497.v7b_061a_a_de65d
gson-api:2.13.2-173.va_a_092315913c
handy-uri-templates-2-api:2.1.8-38.vcea_5d521d5f3
htmlpublisher:427
instance-identity:203.v15e81a_1b_7a_38
ionicons-api:94.vcc3065403257
jackson2-api:2.20.1-423.v13951f6b_6532
jakarta-activation-api:2.1.4-1
jakarta-mail-api:2.1.5-1
jakarta-xml-bind-api:4.0.6-12.vb_1833c1231d3
javadoc:354.vee1a_660b_4990
javax-activation-api:1.2.0-8
javax-mail-api:1.6.2-11
jaxb:2.3.9-143.v5979df3304e6
jdk-tool:83.v417146707a_3d
jenkins-design-language:1.27.25
jjwt-api:0.11.5-120.v0268cf544b_89
job-dsl:1.93
joda-time-api:2.14.0-177.vd7e9347b_e7d5
jquery3-api:3.7.1-619.vdb_10e002501a_
jsch:0.2.16-95.v3eecb_55fa_b_78
json-api:20251224-185.v0cc18490c62c
json-path-api:2.10.0-202.va_9cc16c1e476
jsoup:1.22.1-76.v9cdb_2456c0e3
junit:1369.v15da_00283f06
kubernetes:4423.vb_59f230b_ce53
kubernetes-client-api:7.3.1-256.v788a_0b_787114
kubernetes-credentials:207.v492f58828b_ed
ldap:807.v7d7de30930cf
lockable-resources:1438.v3c0f8c9e2060
mailer:525.v2458b_d8a_1a_71
mapdb-api:1.0.9-44.va_1e1310c9118
matrix-auth:3.2.9
matrix-project:870.v9db_fcfc2f45b_
maven-plugin:3.27
metrics:4.2.37-494.v06f9a_939d33a_
mina-sshd-api-common:2.16.0-167.va_269f38cc024
mina-sshd-api-core:2.16.0-167.va_269f38cc024
nodejs:1.6.6
oauth-credentials:0.657.v7d8dd90b_0382
okhttp-api:4.12.0-195.vc02552c04ffd
oss-symbols-api:442.v99039087229b_
pam-auth:1.12
pipeline-build-step:584.vdb_a_2cc3a_d07a_
pipeline-github-lib:65.v203688e7727e
pipeline-graph-analysis:245.v88f03631a_b_21
pipeline-groovy-lib:787.ve2fef0efdca_6
pipeline-input-step:540.v14b_100d754dd
pipeline-maven:1611.v6a_00c04177b_b_
pipeline-maven-api:1611.v6a_00c04177b_b_
pipeline-milestone-step:138.v78ca_76831a_43
pipeline-model-api:2.2277.v00573e73ddf1
pipeline-model-definition:2.2277.v00573e73ddf1
pipeline-model-extensions:2.2277.v00573e73ddf1
pipeline-rest-api:2.39
pipeline-stage-step:322.vecffa_99f371c
pipeline-stage-tags-metadata:2.2277.v00573e73ddf1
pipeline-stage-view:2.39
plain-credentials:199.v9f8e1f741799
plugin-util-api:6.1192.v30fe6e2837ff
prism-api:1.30.0-701.vf8f8f1f3fd55
pubsub-light:1.19
resource-disposer:0.25
run-condition:276.v97298f3a_cd51
scm-api:724.v7d839074eb_5c
script-security:1385.v7d2d9ec4d909
snakeyaml-api:2.5-143.v93b_c004f89de
sse-gateway:1.28
ssh-agent:386.v36cc0c7582f0
ssh-credentials:361.vb_f6760818e8c
ssh-slaves:3.1096.v0b_cc466e4323
sshd:3.374.v19b_d59ce6610
structs:362.va_b_695ef4fdf9
subversion:1303.vcfd9679fb_c12
support-core:1801.v76b_389d2deec
thinBackup:2.1.3
timestamper:1.30
token-macro:477.vd4f0dc3cb_cf1
trilead-api:2.284.v1974ea_324382
variant:70.va_d9f17f859e0
warnings-ng:12.9996.va_151fb_d6d757
workflow-aggregator:608.v67378e9d3db_1
workflow-api:1384.vdc05a_48f535f
workflow-basic-steps:1098.v808b_fd7f8cf4
workflow-cps:4254.v0c8e228524ea_
workflow-durable-task-step:1464.v2d3f5c68f84c
workflow-job:1571.vb_423c255d6d9
workflow-multibranch:821.vc3b_4ea_780798
workflow-scm-step:466.va_d69e602552b_
workflow-step-api:710.v3e456cc85233
workflow-support:1010.vb_b_39488a_9841
ws-cleanup:0.49
Does anyone have any idea what could be the issue here? I am at my wits end troubleshooting the issue.