gitSCM plug-in mysteriously stopped authenticating

Using Jenkins
1

2 days ago I noticed my daily system test jobs started failing due to being unable to clone my jenkins script repo. I am using credentials for a bot user, that are valid. I can log into the bots github user account no problem, but using the following checkout stage…

        stage('Pull jenkins scripts') {
            steps {
                checkout scmGit(branches: [[name: '*/develop']], extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'jenkins']], userRemoteConfigs: [[credentialsId: 'company-github-creds', url: 'https://github.com/company/jenkins.git']])
            }
        }

results in the following error

[Pipeline] stage
[Pipeline] { (Pull jenkins scripts)
[Pipeline] checkout
The recommended git tool is: NONE
using credential company-github-creds
Fetching changes from the remote Git repository
 > git rev-parse --resolve-git-dir /home/jenkins/workspace/System Testing/Generic Soak Test/jenkins/.git # timeout=10
 > git config remote.origin.url https://github.com/company/jenkins.git # timeout=10
Fetching upstream changes from https://github.com/company/jenkins.git
 > git --version # timeout=10
 > git --version # 'git version 2.17.1'
using GIT_ASKPASS to set credentials company GitHub bot user credentials
 > git fetch --tags --progress -- https://github.com/company/jenkins.git +refs/heads/*:refs/remotes/origin/* # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://github.com/company/jenkins.git
	at PluginClassLoader for git//hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:997)
	at PluginClassLoader for git//hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1239)
	at PluginClassLoader for git//hudson.plugins.git.GitSCM._checkout(GitSCM.java:1310)
	at PluginClassLoader for git//hudson.plugins.git.GitSCM.checkout(GitSCM.java:1277)
	at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:136)
	at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:101)
	at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:88)
	at PluginClassLoader for workflow-step-api//org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress -- https://github.com/company/jenkins.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: remote: Invalid username or password.

I had upgraded to 2.462.3 on the 12th October. Builds were fine until the 23rd, and now none of my builds that use the gitSCM plugin will authenticate. I am really not sure where the problem lies?

Jenkins setup:
Jenkins: 2.462.3
OS: Linux - 5.10.0-28-cloud-amd64
Java: 17.0.8 - BellSoft (OpenJDK 64-Bit Server VM)

antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
asm-api:9.7.1-97.v4cc844130d97
authentication-tokens:1.119.v50285141b_7e1
authorize-project:1.8.0
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.78.1-248.ve27176eb_46cb_
branch-api:2.1182.va_e67548fb_ed2
build-failure-analyzer:2.5.2
build-name-setter:2.4.3
build-timeout:1.33
build-timestamp:1.0.3
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.2.1
cloudbees-folder:6.955.v81e2a_35c08d3
command-launcher:115.vd8b_301cc15d0
commons-lang3-api:3.17.0-84.vb_b_938040b_078
commons-text-api:1.12.0-129.v99a_50df237f7
conditional-buildstep:1.4.3
credentials:1384.vf0a_2ed06f9c6
credentials-binding:681.vf91669a_32e45
display-url-api:2.204.vf6fddd8a_8b_e9
docker-commons:443.v921729d5611d
docker-workflow:580.vc0c340686b_54
durable-task:577.v2a_8a_4b_7c0247
echarts-api:5.5.1-3
eddsa-api:0.3.0-4.v84c6f0f4969e
email-ext:1844.v3ea_a_b_842374a_
font-awesome-api:6.6.0-2
git:5.5.2
git-client:5.0.0
git-parameter:0.9.19
git-tag-message:1.7.1
github:1.40.0
github-api:1.321-468.v6a_9f5f2d5a_7e
github-branch-source:1807.v50351eb_7dd13
github-oauth:597.ve0c3480fcb_d0
google-container-registry-auth:0.3
google-metadata-plugin:0.5
google-oauth-plugin:1.330.vf5e86021cb_ec
google-storage-plugin:1.360.v6ca_38618b_41f
gson-api:2.11.0-85.v1f4e87273c33
instance-identity:201.vd2a_b_5a_468a_a_6
ionicons-api:74.v93d5eb_813d5f
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javax-activation-api:1.2.0-7
javax-mail-api:1.6.2-10
jaxb:2.3.9-1
jdk-tool:80.v8a_dee33ed6f0
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
joda-time-api:2.13.0-93.v9934da_29b_a_e9
jquery3-api:3.7.1-2
json-api:20240303-101.v7a_8666713110
json-path-api:2.9.0-118.v7f23ed82a_8b_8
junit:1307.vdd5b_2646279e
mailer:488.v0c9639c1a_eb_3
matrix-auth:3.2.3
matrix-project:839.vff91cd7e3a_b_2
metrics:4.2.21-451.vd51df8df52ec
mina-sshd-api-common:2.14.0-133.vcc091215a_358
mina-sshd-api-core:2.14.0-133.vcc091215a_358
oauth-credentials:0.653.v14cf2088e950
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
parameterized-trigger:806.vf6fff3e28c3e
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-github:2.8-159.09e4403bc62f
pipeline-githubnotify-step:49.vf37bf92d2bc8
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-groovy-lib:744.v5b_556ee7c253
pipeline-input-step:495.ve9c153f6067b_
pipeline-model-api:2.2214.vb_b_34b_2ea_9b_83
pipeline-model-definition:2.2214.vb_b_34b_2ea_9b_83
pipeline-model-extensions:2.2214.vb_b_34b_2ea_9b_83
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2214.vb_b_34b_2ea_9b_83
pipeline-stage-view:2.34
plain-credentials:183.va_de8f1dd5a_2b_
plugin-util-api:5.1.0
repo:1.16.0
resource-disposer:0.24
run-condition:1.7
scm-api:696.v778d637b_a_762
script-security:1365.v4778ca_84b_de5
slack:751.v2e44153c8fe1
snakeyaml-api:2.3-123.v13484c65210a_
ssh-credentials:343.v884f71d78167
sshd:3.330.vc866a_8389b_58
structs:338.v848422169819
swarm:3.47
thinBackup:2.1.1
timestamper:1.27
token-macro:400.v35420b_922dcb_
trilead-api:2.147.vb_73cc728a_32e
uno-choice:2.8.4
variant:60.v7290fc0eb_b_cd
workflow-api:1336.vee415d95c521
workflow-basic-steps:1058.vcb_fc1e3a_21a_9
workflow-cps:3975.v567e2a_1ffa_22
workflow-durable-task-step:1371.vb_7cec8f3b_95e
workflow-job:1436.vfa_244484591f
workflow-multibranch:795.ve0cb_1f45ca_9a_
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:678.v3ee58b_469476
workflow-support:930.vf51d22b_ce488
ws-cleanup:0.47

2

Not sure what the issue is but there is a get around for anyone else that has encountered this issue. Use the bots github token…

        stage('Pull jenkins scripts') {
            steps {
                withCredentials([string(credentialsId: 'github-token', variable: 'GITHUB_TOKEN'), string(credentialsId: 'github-bot-user', variable: 'GITHUB_USER')]) {
                    dir("jenkins") {
                        git url: "https://${GITHUB_USER}:${GITHUB_TOKEN}@github.com/company/jenkins.git", branch: 'develop'
                    }
                }
            }
        }

And when you want to pull from a tag instead of a branch

            steps {
                withCredentials([string(credentialsId: 'github-token', variable: 'GITHUB_TOKEN'), string(credentialsId: 'github-bot-user', variable: 'GITHUB_USER')]) {
                    dir("jenkins") {
                        checkout([
                            $class: 'GitSCM',
                            branches: [[name: "refs/tags/v1.0.0"]],
                            doGenerateSubmoduleConfigurations: false,
                            extensions: [],
                            submoduleCfg: [],
                            userRemoteConfigs: [[
                                url: "https://${GITHUB_USER}:${GITHUB_TOKEN}@github.com/company/jenkins.git"
                            ]]
                        ])
                    }