Hello, I hope this is the correct place to ask my question and that I can get some help from the community. I have an intermittent issue and can’t pinpoint where the issues is.
In short, I’m running a few multi-branch pipelines that are activated via a WebHook from GitHub. I have configured a GitHub App, which activates the WebHook for the connected projects. In Jenkins, I have added the necessary credentials and everything works great…then it doesn’t.
Every now and then I get an “Invalid username or password” error with a status code of 128. The interesting thing is (and you’ll see this in the logs below) that the connection to GitHub is fine because it successfully downloads the Jenkinsfiles. When it does fail, I can immediately re-run the job and it works perfectly. This will continue to work perfectly until a few hours later when it will fail again. This more often than not fails on new branches. I created a test app in GitHub and hooked it up to Jenkins and it failed. Here is the output from the log (I have redacted some info I believe to be private with *****):
Push event to branch main
11:04:31 Connecting to https://api.github.com using ******/****** (GitHub App: jenkins-app-credentials)
Obtained Jenkinsfile from *****
[Pipeline] Start of Pipeline
[Pipeline] node
Running on ***** in E:\tools\jenkins-agent\workspace\Test_Multibranch_Pipeline_main
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
The recommended git tool is: NONE
using credential jenkins-app-credentials
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository https://github.com/*****/TestRepo.git
> git.exe init E:\tools\jenkins-agent\workspace\Test_Multibranch_Pipeline_main # timeout=10
Fetching upstream changes from https://github.com/*****/TestRepo.git
> git.exe --version # timeout=10
> git --version # 'git version 2.43.0.windows.1'
using GIT_ASKPASS to set credentials GitHub App: jenkins-app-credentials
> git.exe fetch --no-tags --force --progress -- https://github.com/*****/TestRepo.git +refs/heads/main:refs/remotes/origin/main # timeout=10
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "git.exe fetch --no-tags --force --progress -- https://github.com/*****/TestRepo.git +refs/heads/main:refs/remotes/origin/main" returned status code 128:
stdout:
stderr: remote: Invalid username or password.
fatal: Authentication failed for 'https://github.com/*****/TestRepo.git/'
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2842)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:635)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:871)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:377)
at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125)
at java.base/java.lang.Thread.run(Thread.java:842)
Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to web-test-104
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1787)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
at hudson.remoting.Channel.call(Channel.java:1003)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:153)
at jdk.internal.reflect.GeneratedMethodAccessor762.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:138)
at jdk.proxy52/jdk.proxy52.$Proxy143.execute(Unknown Source)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1222)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1305)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
... 1 more
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: Error cloning remote repo 'origin'
GitHub has been notified of this commit’s build result
Finished: FAILURE
As you can see from the above log, it has successfully downloaded the Jenkinsfile, however I get an “Invalid username or password” and the rest fails.
I’m not sure if this is relevant or will help but I’ve included below the contents of my Jenkinsfile. You can see from the logs that it hasn’t even processed the “Init” stage, given I can’t see any “Init started” in the log.
pipeline {
agent any
options {
buildDiscarder logRotator(artifactDaysToKeepStr: '', artifactNumToKeepStr: '5', daysToKeepStr: '', numToKeepStr: '5')
disableConcurrentBuilds();
}
stages {
stage('Init') {
steps {
script {
println "Init started"
def currentDirectory = pwd()
println "Current working directory: ${currentDirectory}"
def branchName = env.GIT_BRANCH
println "Current branch name: ${branchName}"
}
}
}
stage('Checkout') {
steps {
// Check out your source code from the GitHub repository
checkout([$class: 'GitSCM', branches: [[name: env.GIT_BRANCH]], doGenerateSubmoduleConfigurations: false, extensions: [], userRemoteConfigs: [[url: 'https://github.com/Auto-IT-Australia/TestRepo', credentialsId: 'jenkins-auto-it-australia']]])
}
}
}
}
I’m very confused as to why this is failing and have no idea even where to start looking. Any help you can provide will be valuable and very much appreciated.
Jenkins setup:
Jenkins: 2.426.3
OS: Windows Server 2022 - 10.0
Java: 17.0.9 - Oracle Corporation (Java HotSpot™ 64-Bit Server VM)
ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
bootstrap5-api:5.3.2-3
bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9
branch-api:2.1148.vce12cfcdf090
build-timeout:1.32
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.0.2
cloudbees-folder:6.858.v898218f3609d
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.11.0-95.v22a_d30ee5d36
credentials:1319.v7eb_51b_3a_c97b_
credentials-binding:657.v2b_19db_7d6e6d
display-url-api:2.200.vb_9327d658781
durable-task:550.v0930093c4b_a_6
echarts-api:5.4.3-2
email-ext:2.104
font-awesome-api:6.5.1-2
git:5.2.1
git-client:4.6.0
github:1.38.0
github-api:1.318-461.v7a_c09c9fa_d63
github-branch-source:1772.va_69eda_d018d4
gradle:2.10
gson-api:2.10.1-15.v0d99f670e0a_7
instance-identity:185.v303dc7c645f9
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.16.1-373.ve709c6871598
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.9-1
jjwt-api:0.11.5-77.v646c772fddb_0
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery3-api:3.7.1-1
json-path-api:2.9.0-33.v2527142f2e1d
junit:1259.v65ffcef24a_88
ldap:711.vb_d1a_491714dc
mailer:463.vedf8358e006b_
matrix-auth:3.2.1
matrix-project:822.824.v14451b_c0fd42
mina-sshd-api-common:2.12.0-90.v9f7fb_9fa_3d3b_
mina-sshd-api-core:2.12.0-90.v9f7fb_9fa_3d3b_
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
pam-auth:1.10
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:704.vc58b_8890a_384
pipeline-input-step:477.v339683a_8d55e
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2175.v76a_fff0a_2618
pipeline-model-definition:2.2175.v76a_fff0a_2618
pipeline-model-extensions:2.2175.v76a_fff0a_2618
pipeline-rest-api:2.34
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2175.v76a_fff0a_2618
pipeline-stage-view:2.34
pipeline-utility-steps:2.16.2
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:3.8.0
resource-disposer:0.23
scm-api:683.vb_16722fb_b_80b_
script-security:1326.vdb_c154de8669
snakeyaml-api:2.2-111.vc6598e30cc65
ssh-credentials:308.ve4497b_ccd8f4
ssh-slaves:2.948.vb_8050d697fec
structs:337.v1b_04ea_4df7c8
timestamper:1.26
token-macro:400.v35420b_922dcb_
trilead-api:2.133.vfb_8a_7b_9c5dd1
variant:60.v7290fc0eb_b_cd
workflow-aggregator:596.v8c21c963d92d
workflow-api:1291.v51fd2a_625da_7
workflow-basic-steps:1042.ve7b_140c4a_e0c
workflow-cps:3867.v535458ce43fd
workflow-durable-task-step:1331.vc8c2fed35334
workflow-job:1385.vb_58b_86ea_fff1
workflow-multibranch:773.vc4fe1378f1d5
workflow-scm-step:415.v434365564324
workflow-step-api:657.v03b_e8115821b_
workflow-support:865.v43e78cc44e0d
ws-cleanup:0.45
