Intermittently receiving 401 Unauthorized from api while using a access token (PAT)

Using Jenkins
1

I just updated to 2.462.2-lts and now receive (sometimes it works) 401 unauthorized. Sometimes a build will fail to trigger, other times 401. If the build can be successfully triggered, it can fail downstream when making /api/xml calls (happens most of the time).

I also reproduced this with simple calls using Postman, but not consistently as to be able to pinpoint the actual issue. As an example, a scheduled job initiated overnight that received 401 after successfully initiating build. This morning I used postman (with same API token) to hit /api/xml and also received 401 multiple times (same payload repeated). I logged out of the UI (that was left open overnight) tried postman again, and received 200 success. A little while later starting receiving 401 again.

I have been successfully integrating with the API using a Personal Access Token (PAT) for 2-3 years on version 2.375.3-lts without any issues. The PAT is also being used by CI/CD tools. UI is integrated with SAML so token has to be used.

Here is a screenshot from the CICD tool where 7 jobs were successful, but the last one received 401 (the build submitted but 401 was received while waiting for completion).

image
image1222×838 56.8 KB

As you can see from the build numbers, we have executed 500+ builds (99% successful) over a couple years.

Any thoughts on next steps?

Jenkins setup:
Jenkins: 2.462.2
OS: Linux - 5.15.0-1059-azure
Java: 17.0.12 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)

ant:511.v0a_a_1a_334f41b_
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
asm-api:9.7-33.v4d23ef79fcc8
authentication-tokens:1.119.v50285141b_7e1
azure-credentials:312.v0f3973cd1e59
azure-sdk:174.va_89c1df897d2
blueocean-commons:1.27.16
blueocean-rest:1.27.16
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.78.1-248.ve27176eb_46cb_
branch-api:2.1178.v969d9eb_c728e
build-timeout:1.33
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.2.1
cloudbees-folder:6.951.v5f91d88d76b_b_
command-launcher:115.vd8b_301cc15d0
commons-compress-api:1.26.1-2
commons-lang3-api:3.17.0-84.vb_b_938040b_078
commons-text-api:1.12.0-129.v99a_50df237f7
config-file-provider:978.v8e85886ffdc4
copyartifact:749.vfb_dca_a_9b_6549
credentials:1378.v81ef4269d764
credentials-binding:681.vf91669a_32e45
display-url-api:2.204.vf6fddd8a_8b_e9
docker-commons:443.v921729d5611d
docker-workflow:580.vc0c340686b_54
durable-task:577.v2a_8a_4b_7c0247
echarts-api:5.5.1-1
eddsa-api:0.3.0-4.v84c6f0f4969e
email-ext:1814.v404722f34263
font-awesome-api:6.6.0-2
git:5.5.1
git-client:5.0.0
github:1.40.0
github-api:1.321-468.v6a_9f5f2d5a_7e
github-branch-source:1797.v86fdb_4d57d43
gradle:2.13
gson-api:2.11.0-41.v019fcf6125dc
instance-identity:185.v303dc7c645f9
ionicons-api:74.v93d5eb_813d5f
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javax-activation-api:1.2.0-7
javax-mail-api:1.6.2-10
jaxb:2.3.9-1
jdk-tool:80.v8a_dee33ed6f0
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
joda-time-api:2.13.0-85.vb_64d1c2921f1
jquery3-api:3.7.1-2
json-api:20240303-41.v94e11e6de726
json-path-api:2.9.0-58.v62e3e85b_a_655
junit:1300.v03d9d8a_cf1fb_
kubernetes:4288.v1719f9d0c854
kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
kubernetes-credentials:190.v03c305394deb_
ldap:725.v3cb_b_711b_1a_ef
mailer:472.vf7c289a_4b_420
matrix-auth:3.2.2
matrix-project:832.va_66e270d2946
metrics:4.2.21-451.vd51df8df52ec
mina-sshd-api-common:2.13.2-125.v200281b_61d59
mina-sshd-api-core:2.13.2-125.v200281b_61d59
oic-auth:4.354.v321ce67a_1de8
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
pam-auth:1.11
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-github-lib:61.v629f2cc41d83
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-groovy-lib:730.ve57b_34648c63
pipeline-input-step:495.ve9c153f6067b_
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2214.vb_b_34b_2ea_9b_83
pipeline-model-definition:2.2214.vb_b_34b_2ea_9b_83
pipeline-model-extensions:2.2214.vb_b_34b_2ea_9b_83
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2214.vb_b_34b_2ea_9b_83
pipeline-stage-view:2.34
pipeline-utility-steps:2.17.0
plain-credentials:183.va_de8f1dd5a_2b_
plugin-util-api:4.1.0
popper2-api:2.11.6-5
resource-disposer:0.23
scm-api:696.v778d637b_a_762
script-security:1362.v67dc1f0e1b_b_3
snakeyaml-api:2.3-123.v13484c65210a_
ssh-credentials:343.v884f71d78167
ssh-slaves:2.973.v0fa_8c0dea_f9f
sshd:3.330.vc866a_8389b_58
strict-crumb-issuer:2.1.1
structs:338.v848422169819
timestamper:1.27
token-macro:400.v35420b_922dcb_
trilead-api:2.147.vb_73cc728a_32e
variant:60.v7290fc0eb_b_cd
windows-azure-storage:419.v4046cd70d2e3
workflow-aggregator:600.vb_57cdd26fdd7
workflow-api:1336.vee415d95c521
workflow-basic-steps:1058.vcb_fc1e3a_21a_9
workflow-cps:3964.v0767b_4b_a_0b_fa_
workflow-durable-task-step:1371.vb_7cec8f3b_95e
workflow-job:1436.vfa_244484591f
workflow-multibranch:795.ve0cb_1f45ca_9a_
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:678.v3ee58b_469476
workflow-support:926.v9f4f9b_b_98c19
ws-cleanup:0.46

2

I am yet to determine what has to happen for the api request to go from status 200 OK to 401 Unauthorized, but once that shift occurs, I can repeatedly use postman and receive nothing but 401 Unauthorized no matter how much time passes or number of requests made.

If I click " Terminate All Sessions" under my user account the 401 Unauthorized from postman requests started returning 200 OK again.

3

It took 20-30 mins for the responses to go from 200 back to 401. I terminated sessions again and responses were successful again. The PAT seems tied to the UI sessions somehow. I can’t find any documentation regarding this. All documentation I can find talk about PATs having no expiration.

This definitely seems like a bug.

4

I don’t see any problems on my side with PATs. I’m using the same token since long time to trigger builds via the REST API without any problems.
I’m also on 2.462.x

5

Thanks for responding Markus. Are you also using SAML?

6

No, on that instance I’m using Active Directory plugin

7

Markus, do you have Strict Crumb Issuer enabled?

Here is how I have it configured (has not changed since upgrade)

image
image1141×697 23.7 KB

All builds failed to initiate this morning:

image
image1224×780 50.4 KB

8

no, I’m using the default crumb issuer

9

I have continued troubleshooting and hope I have found the issue - I am testing.

I switched from strict crumb issuer to the default issuer but still had the issue.
I disabled the crumbs (CSRF) entirely but still had the issue.
I switched from Personal Access Token (PAT) to legacy tokens but still had the issue.

I starting researching the SSO integration. The clarification I need to make from the above posts, is that I’m using OpenId Connect Authentication, and not SAML.

I have found this issue: Allow access using a Jenkins API token without an OIDC Session by mikecirioli · Pull Request #386 · jenkinsci/oic-auth-plugin · GitHub. In summary this is how the plugin was working:
“Currently, when using the oic-auth plugin, if a user explicitly logs out of the IdP then they will now longer be able to perform any sort of action using jenkins api tokens because the oic-auth plugin will reject the request due to the OicSession being null.”

I can’t speak for the reason why they would tie API tokens to sessions to begin with as that seems weird to me, but I’m glad there is now an option to skip the session check. I have checked the checkbox and will monitor the issue.

11

This has solved the issue.