Random '403 Invalid Crumb' messages through-out Jenkins

bt-abarber · September 19, 2025, 10:29am

A couple of months ago I started to experience random 403 “no valid crumb” errors through-out when using Jenkins. The errors show up particularly frequently when trying to replay jobs but they seem to happen on just about any page depending on the embedded content. I’ve updated Jenkins at least once since this started happening and numerous plug-ins have been updated as well. This current Jenkins instance had been working for a year or more without issue until just recently and I had a similar instance I used for numerous years before that.

The problems aren’t limited to a particular browser either (happens for sure in Chrome/Firefox)

The most common instance of the error is after submitting a form to start a job (this is a rebuild but it happens when building a new job fresh also):

Sometimes the errors show up below various input fields:

See post below for screenshot as I can apparently only include one per post

Simply hitting the browser back button and retrying often works, but sometimes it doesn’t. I’ve had it fail upwards of a dozen times in a row (usually I get frustrated and move on to something else for bit). This does not seem to happen anywhere but though the web UI, scripted access works without issue.

When viewing console logs steaming in for various running jobs the browser will also sometimes just get stuck in some kind of rapid refresh loop trying to load the page content, the header and side bar display/disappear rapidly and the title in the tab disappears/re-appears rapidly.

I’ve also noticed that I (again sometimes, if I simply refresh the page it goes away) get a notice that reverse proxy is broken - Note: I do not have a reverse proxy configured, it’s a pretty vanilla install of Jenkins on a Windows Server instance.

See post below for screenshot as I can apparently only include one per post

I have worked with our IT department trying to figure this out, they can’t see anything on their end that would be impacting things. While monitoring traffic to/from my laptop and the Jenkins instance, as well as the Jenkins instance to the internet, they’re not seeing any blocked/suspect traffic. I’m told they also looked at the firewall and can’t see that would impact anything. After several attempts to resolve the matter they requested I open this ticket and provide as much information as I can.

Coles Notes:

Occasional problem started happening on a previously functioning, quite basic install of Jenkins.
Occurs on numerous pages within Jenkins when trying to run/re-run jobs
Does not happen for scripted access, only within the UI.
Not limited to one particular user
Not limited to one particular browser
Occasional message about reverse proxy being broken, no reverse proxy configured.
No obvious traffic being limited on a firewall/network level

**

Jenkins setup:**

Jenkins: 2.516.2
OS: Windows Server 2022 - 10.0
Java: 21.0.1 - Oracle Corporation (Java HotSpot™ 64-Bit Server VM)

ant:518.v8d8dc7945eca_
antisamy-markup-formatter:173.v680e3a_b_69ff3
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
asm-api:9.8-163.vb_2a_96d3f9c3c
bootstrap5-api:5.3.7-860.v1251c115c90c
bouncycastle-api:2.30.1.81-264.v95c79c0e772c
branch-api:2.1244.vf95c81f1641c
build-timeout:1.38
caffeine-api:3.2.2-178.v353b_8428ed56
checks-api:373.vfe7645102093
cloudbees-folder:6.1040.v8a_e6330a_54e3
commons-compress-api:1.28.0-1
commons-lang3-api:3.18.0-98.v3a_674c06072d
commons-text-api:1.14.0-194.v804a_dc3a_1b_d8
copyartifact:770.va_6c69e063442
credentials:1419.v2337d1ceceef
credentials-binding:702.vfe613e537e88
data-tables-api:2.3.3-1383.va_5607a_a_3f3c2
display-url-api:2.217.va_6b_de84cc74b_
durable-task:595.ve87b_f1318d67
echarts-api:6.0.0-1146.v5c8f3b_8f0573
eddsa-api:0.3.0.1-19.vc432d923e5ee
email-ext:1925.v1598902b_58dd
font-awesome-api:7.0.0-851.vd1feb_218a_a_63
git:5.7.0
git-client:6.3.3
github:1.45.0
github-api:1.330-492.v3941a_032db_2a_
github-branch-source:1848.v42f74f7f4500
gradle:2.16.1149.v711b_998b_0532
gson-api:2.13.1-153.vb_3d0c48a_a_b_4a_
instance-identity:203.v15e81a_1b_7a_38
ionicons-api:94.vcc3065403257
jackson2-api:2.20.0-411.v6ef8fdee4fe9
jakarta-activation-api:2.1.3-2
jakarta-mail-api:2.1.3-3
javax-activation-api:1.2.0-8
javax-mail-api:1.6.2-11
jaxb:2.3.9-133.vb_ec76a_73f706
jjwt-api:0.11.5-120.v0268cf544b_89
joda-time-api:2.14.0-149.v1c3ce991d1b_9
jquery3-api:3.7.1-594.vb_3864f326cf0
json-api:20250517-173.v596efb_962a_31
json-path-api:2.9.0-190.veefca_05d5477
jsoup:1.21.2-66.v6ea_38164b_8a_2
junit:1355.v45e2ea_65863c
ldap:780.vcb_33c9a_e4332
lockable-resources:1412.v3f305a_fb_a_117
mailer:522.va_995fa_cfb_8b_d
matrix-auth:3.2.8
matrix-project:856.v4c352b_3a_b_23e
mina-sshd-api-common:2.16.0-167.va_269f38cc024
mina-sshd-api-core:2.16.0-167.va_269f38cc024
okhttp-api:4.11.0-189.v976fa_d3379d6
pam-auth:1.12
pipeline-build-step:571.v08a_fffd4b_0ce
pipeline-github-lib:65.v203688e7727e
pipeline-graph-analysis:241.vc3d48fb_b_2582
pipeline-groovy-lib:752.vdddedf804e72
pipeline-input-step:534.v352f0a_e98918
pipeline-milestone-step:138.v78ca_76831a_43
pipeline-model-api:2.2265.v140e610fe9d5
pipeline-model-definition:2.2265.v140e610fe9d5
pipeline-model-extensions:2.2265.v140e610fe9d5
pipeline-rest-api:2.38
pipeline-stage-step:322.vecffa_99f371c
pipeline-stage-tags-metadata:2.2265.v140e610fe9d5
pipeline-stage-view:2.38
pipeline-utility-steps:2.19.0
plain-credentials:199.v9f8e1f741799
plugin-util-api:6.1167.v022176c7e0ca_
rebuild:338.va_0a_b_50e29397
resource-disposer:0.25
scm-api:707.v749f968369d4
script-security:1378.vf25626395f49
snakeyaml-api:2.3-125.v4d77857a_b_402
ssh-credentials:361.vb_f6760818e8c
ssh-slaves:3.1071.v0d059c7b_c555
structs:353.v261ea_40a_80fb_
timestamper:1.30
token-macro:477.vd4f0dc3cb_cf1
trilead-api:2.209.v0e69b_c43c245
variant:70.va_d9f17f859e0
workflow-aggregator:608.v67378e9d3db_1
workflow-api:1384.vdc05a_48f535f
workflow-basic-steps:1079.vce64b_a_929c5a_
workflow-cps:4183.v94b_6fd39da_c1
workflow-durable-task-step:1452.v0ee719c104a_7
workflow-job:1540.v295eccc9778f
workflow-multibranch:811.vcd33d074c2a_0
workflow-scm-step:437.v05a_f66b_e5ef8
workflow-step-api:706.v518c5dcb_24c0
workflow-support:976.vb_d9493c2eb_09
ws-cleanup:0.49

bt-abarber · September 19, 2025, 10:30am

bt-abarber · September 19, 2025, 10:31am

poddingue · September 19, 2025, 1:02pm

Hello and welcome to this community, @bt-abarber!

What you’ve described makes me think of Jenkins’ Cross-Site Request Forgery (CSRF) protection and session cookie handling.
Even when there’s no reverse proxy, a small mismatch or browser-side issue can break crumb validation and cause those “Invalid crumb” or “403” errors.

Here’s a checklist to try and fix it:

Session / Crumb Troubleshooting Checklist

1. Validate Jenkins Base URL

Go to Manage Jenkins (top right ) → System
Make sure Jenkins URL is exactly the same as how you access it (protocol, domain, and port).
- http://localhost:8080
- https://ci.example.com

This value is used in cookies and CSRF crumb generation.

2. Reset Browser State

Clear cookies, cache, and site data for your Jenkins URL.
Try a private/incognito window.
Temporarily disable all browser extensions (especially ad-blockers, privacy blockers, and password managers).

3. Ensure Single Entry Point

Always use one consistent hostname to access Jenkins (not mixing localhost, IP, and DNS names).
If switching URLs, cookies may get mixed up and cause invalid crumbs.

4. Check Server Clock

Ensure both the server and your workstation clocks are accurate.
- Misaligned clocks break session cookies and CSRF validation.

5. Look for External Interference

Antivirus/firewall/enterprise network tools can strip or block cookies/headers.
Test from a different network or machine to isolate the issue.

6. Check Logs

In jenkins.log, look for:
- Invalid crumb
- Rejected due to missing or bad crumb
- Cookie rejected
  These messages could confirm CSRF-related problems.

7. Eliminate Local Corruption

Create a new Jenkins user and log in with it to see if the issue persists.
As a last resort, try running Jenkins on a different port or with a fresh $JENKINS_HOME to rule out config corruption.

Summary

Root cause is often: URL mismatch, cookies corruption, or session misalignment.
Fixing the Jenkins URL + clearing cookies + toggling CSRF settings should solve it in lots of cases.

Here’s a small Jenkins Script Console snippet you can run to quickly inspect your CSRF (crumb) configuration and security status:

Groovy Diagnostic Script

import jenkins.model.Jenkins
import hudson.security.csrf.CrumbIssuer
import jenkins.security.s2m.AdminWhitelistRule

def j = Jenkins.instance

println "=== Jenkins Security & CSRF Diagnostics ==="
println "Jenkins URL: ${j.getRootUrl() ?: '(not set)'}"
println "Security Realm: ${j.getSecurityRealm()?.class?.name ?: 'none'}"
println "Authorization Strategy: ${j.getAuthorizationStrategy()?.class?.name ?: 'none'}"

CrumbIssuer ci = j.getCrumbIssuer()
if (ci) {
    println "Crumb issuer: ${ci.class.name}"
    println "  - Crumb request field: ${ci.getCrumbRequestField()}"
    println "  - Sample crumb: ${ci.getCrumb()}"
} else {
    println "Crumb issuer: NONE (CSRF protection is disabled)"
}

def whitelist = j.getInjector().getInstance(AdminWhitelistRule)
println "Master-to-Agent Access Control: ${whitelist.getMasterKillSwitch() ? 'DISABLED' : 'ENABLED'}"

println "=== End ==="

How to Use

Go to Manage Jenkins → Script Console.
Paste the script above.
Click Run.

What It Shows

Configured Jenkins URL (important for cookie/crumb domains)
Security realm & authorization strategy
Which crumb issuer is active
Status of controller-to-Agent Access Control (just for context)

bt-abarber · September 19, 2025, 5:54pm

Thanks for the great reply - fortunately I’ve been through some of these things already in my efforts to self-diagnose/resolve.

1. Validate Jenkins Base URL

Done, this was actually one of the first things I did (as was suggested when searching Google results) when I tried to troubleshoot when it tried happening the first time.

2. Reset Browser State

Done, numerous browsers - one freshly installed, incognito modes, from different accounts on numerous systems has resulted in the same issue.

3. Ensure Single Entry Point

Done.

4. Check Server Clock

Done, was also one thing suggested by Google search results.

5. Look for External Interference

Also done, and this is where things go sideways.

Accessing Jenkins from systems on the same vSphere instance as where the Jenkins instance is hosted can do so without issue. My co-workers and I connecting using our local machines and a VPN to gain access is where we have issues.

6. Check Logs

Small sample in the event the errors provide some insight that isn’t standing out to me:

Found invalid crumb 1b977c68fe51cd375d501038ff49784b37ae96d1f8fe6c18713d1abd633a9fd3. If you are calling this URL with a script, please use the API Token instead. More information: https://www.jenkins.io/redirect/crumb-cannot-be-used-for-script

Sep 19, 2025 2:34:38 PM WARNING hudson.security.csrf.CrumbFilter doFilter

No valid crumb was included in request for /manage/descriptorByName/hudson.plugins.gradle.injection.InjectionConfig/checkAccessKeyCredentialId by admin. Returning 403.

Sep 19, 2025 2:34:38 PM WARNING hudson.security.csrf.CrumbFilter doFilter

Found invalid crumb 1b977c68fe51cd375d501038ff49784b37ae96d1f8fe6c18713d1abd633a9fd3. If you are calling this URL with a script, please use the API Token instead. More information: https://www.jenkins.io/redirect/crumb-cannot-be-used-for-script

Sep 19, 2025 2:34:38 PM WARNING hudson.security.csrf.CrumbFilter doFilter

No valid crumb was included in request for /manage/descriptorByName/jenkins.model.ProjectNamingStrategy$PatternProjectNamingStrategy/checkNamePattern by admin. Returning 403.

Sep 19, 2025 2:34:38 PM WARNING hudson.security.csrf.CrumbFilter doFilter

Found invalid crumb 1b977c68fe51cd375d501038ff49784b37ae96d1f8fe6c18713d1abd633a9fd3. If you are calling this URL with a script, please use the API Token instead. More information: URL REMOVED BECAUSE FORUM YELLS AT ME FOR HAVING TOO MANY URLS

Sep 19, 2025 2:34:38 PM WARNING hudson.security.csrf.CrumbFilter doFilter

No valid crumb was included in request for /manage/descriptorByName/hudson.triggers.SCMTrigger/checkPollingThreadCount by admin. Returning 403.

Sep 19, 2025 2:34:38 PM WARNING hudson.security.csrf.CrumbFilter doFilter

Found invalid crumb 1b977c68fe51cd375d501038ff49784b37ae96d1f8fe6c18713d1abd633a9fd3. If you are calling this URL with a script, please use the API Token instead. More information: URL REMOVED BECAUSE FORUM YELLS AT ME FOR HAVING TOO MANY URLS

Sep 19, 2025 2:34:38 PM WARNING hudson.security.csrf.CrumbFilter doFilter

No valid crumb was included in request for /manage/descriptorByName/hudson.plugins.gradle.injection.InjectionConfig/fillMavenExtensionRepositoryCredentialIdItems by admin. Returning 403.

Sep 19, 2025 2:44:53 PM WARNING hudson.security.csrf.CrumbFilter doFilter

Found invalid crumb 0ecfdbd87bb8f0b74cd763d5084b8c8b09a74bb4f0debfd2088e1357bedff1ec. If you are calling this URL with a script, please use the API Token instead. More information: URL REMOVED BECAUSE FORUM YELLS AT ME FOR HAVING TOO MANY URLS

Sep 19, 2025 2:44:53 PM WARNING hudson.security.csrf.CrumbFilter doFilter

No valid crumb was included in request for /job/----------------------------/40460/replay/checkScript by admin. Returning 403.

7. Eliminate Local Corruption

I have not tried this yet and I’m unfortunately out of time for the week. I figured I’d post my findings thus far in the event it causes a light-bulb to come on for anyone who may be reading this post

jimklimov · September 21, 2025, 11:43am

Just in case, I also often see these when I have Jenkins pages open and the controller was restarted - its new uptime does not seem to trust some cookies of the old one (would love a fix for that, maybe some local misconfig?) So replays of a job, apply of a config change, following the build queue, etc. do not happen because responses do not come, unless I explicitly refresh the page and get new crumbs. The old (auto-?)login is still trusted at this point though, so I do get my authenticated version of that page after the reload.

Jim

mawinter69 · September 21, 2025, 6:32pm

This is a strong indicator that the problems are related to your network. Not sure but I think the IP address is also part of the crumb calculation. So in case you have a some kind of load balancing in front of the vSpere setup that works with different IPs that might cause the problems. I suggest to enable access logging (if not already done) and check from which IPs the requests are coming

bt-abarber · September 22, 2025, 11:04am

Thanks for the suggestion but this happens most frequently towards the end of the day after I’ve typically been successfully using things for several hours. I know the Jenkins instance hasn’t been restarted during this time. I do understand what you’re getting at though as I’ve logged on the next day, after an automatic reboot of Jenkins, and while it appears I’m logged in I also encounter issues when trying to submit forms or the like until I refresh things.

bt-abarber · September 22, 2025, 11:05am

Innnnnnteresting, a load balancer may actually explain the issues quite well if this is the case. I will attempt your suggestion of enabling access logging and see what kind of results that yields.

bt-abarber · September 24, 2025, 11:18am

Well thank-you again for your suggestion, it ultimately pointed us down the road towards finding a solution.

Ultimately it appears the issue was because of how our VPN (Cloudflare WARP) handles traffic. The access logs showed most requests coming from the same IP however some, randomly, would come from different IPs. This ultimately led us to the following:

Enabling “Proxy Compatibility” seems to have resolved the issue entirely.

I figured I’d update this ticket in the event someone else encounters the same and it helps them.

poddingue · September 24, 2025, 11:36am

Tracking down the issue couldn’t have been easy, but you’ve done an impressive job.
Thank you for your detailed feedback.

Topic		Replies	Views
How to fix 'invalid crumb' warnings? Using Jenkins question	11	28280	October 9, 2023
Upgrade from 2.204.6 to 2.289.3 failed on CSRF Using Jenkins	6	1514	August 10, 2022
Tomcat + Jenkins 2.346.3 CRUMB error on 1st startup Ask a question	10	1384	November 8, 2022
Jenkins works during days and then no jobs can be run anymore on any agent Ask a question	8	1710	November 29, 2022
Jenkins Error Message:- No valid crumb was included in the request, Description:- The server understood the request but refuses to Authorize it Ask a question	1	967	September 5, 2022

Random '403 Invalid Crumb' messages through-out Jenkins

Jenkins setup:**

Session / Crumb Troubleshooting Checklist

1. Validate Jenkins Base URL

2. Reset Browser State

3. Ensure Single Entry Point

4. Check Server Clock

5. Look for External Interference

6. Check Logs

7. Eliminate Local Corruption

Summary

Groovy Diagnostic Script

How to Use

What It Shows

1. Validate Jenkins Base URL

2. Reset Browser State

3. Ensure Single Entry Point

4. Check Server Clock

5. Look for External Interference

6. Check Logs

7. Eliminate Local Corruption

Related topics