TL/DR
I looked at How to debug GIT Checkout Authentication Failure? - #2 by MarkEWaite, Why is Jenkins failing when fetching from git, while the command line isn't? - Stack Overflow, git - Jenkins fails on checkout - Stack Overflow and various other search results.
Error
On one Jenkins node checkout of a bitbucket.org repo fails. On several other nodes, the exact same checkout succeeds. The nodes are connected to the same Jenkins controller.
The x-auth-token is stored in the credential manager on the controller.
hudson.plugins.git.GitException: Command "git fetch --tags --force --progress -- https://bitbucket.org/company/reponame.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
Extra info
- posted the same question on stackoverflow.
- git clone from the command line, using the same credentials, the same linux user it succeeds.
- The same job on a different linux host with the same Jenkins plugin version succeeds.
- The error occurred in a scripted jenkinsfile
- I created a simple free-style job that just does the checkout and it shows the same error, logging below is from that simple job.
I added GIT_TRACE=1 to the environment to get some extra logging.
Succesful log on node company-dev02
Started by user User name
Running as SYSTEM
[EnvInject] - Loading node environment variables.
[EnvInject] - Preparing an environment for the build.
[EnvInject] - Keeping Jenkins system variables.
[EnvInject] - Keeping Jenkins build variables.
[EnvInject] - Injecting as environment variables the properties content
GIT_TRACE=1
[EnvInject] - Variables injected successfully.
[EnvInject] - Injecting contributions.
Building remotely on company-dev02 (developmenthost linuxreleasehost linux latest22_linux_db test22cloud_linux_db test22_linux_db develop23_linux_db develop_linux_db) in workspace /var/lib/jenkins/workspace/cceadm appltest22/xcomp checkout company-dev02
[EnvInject] - Unset unresolved 'BASH_FUNC_which%%' variable.
The recommended git tool is: NONE
using credential tokentokenTOKENTOKEN
Cloning the remote Git repository
Cloning repository https://bitbucket.org/company/reponame.git
> git init /var/lib/jenkins/workspace/cceadm appltest22/xcomp checkout company-dev02 # timeout=10
Fetching upstream changes from https://bitbucket.org/company/reponame.git
> git --version # timeout=10
> git --version # 'git version 2.31.1'
using GIT_ASKPASS to set credentials token TOKENTOKEN
> git fetch --tags --force --progress -- https://bitbucket.org/company/reponame.git +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url https://bitbucket.org/company/reponame.git # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
> git rev-parse refs/remotes/origin/xcomp/22.0/test^{commit} # timeout=10
> git rev-parse xcomp/22.0/test^{commit} # timeout=10
Checking out Revision e7ce93ef07cdaff3ae372dfed57c93ccf3ebbd84 (refs/remotes/origin/xcomp/22.0/test)
> git config core.sparsecheckout # timeout=10
> git checkout -f e7ce93ef07cdaff3ae372dfed57c93ccf3ebbd84 # timeout=10
Commit message: "Merge branch 'user/config/company-812-Cloud-build' into xcomp/22.0/test"
> git rev-list --no-walk e7ce93ef07cdaff3ae372dfed57c93ccf3ebbd84 # timeout=10
18:51:17.933383 git.c:447 trace: built-in: git whatchanged --no-abbrev -M '--format=commit %H%ntree %T%nparent %P%nauthor %aN <%aE> %ai%ncommitter %cN <%cE> %ci%n%n%w(0,4,4)%B' -n 1024 e7ce93ef07cdaff3ae372dfed57c93ccf3ebbd84 ^e7ce93ef07cdaff3ae372dfed57c93ccf3ebbd84
The recommended git tool is: NONE
using credential tokentokenTOKENTOKEN
> git rev-parse e7ce93ef07cdaff3ae372dfed57c93ccf3ebbd84^{commit} # timeout=10
The recommended git tool is: NONE
using credential tokentokenTOKENTOKEN
[GitCheckoutListener] Recording commits of 'git https://bitbucket.org/company/reponame.git'
[GitCheckoutListener] Found previous build 'cceadm appltest22/xcomp checkout company-dev02 #2' that contains recorded Git commits
[GitCheckoutListener] -> Starting recording of new commits since 'e7ce93e'
[GitCheckoutListener] -> Multiple parent commits found - storing latest commit of local merge 'e7ce93e'
[GitCheckoutListener] -> Using parent commit '05cef30' of local merge as starting point
[GitCheckoutListener] -> Storing target branch head '74f329b' (second parent of local merge)
[GitCheckoutListener] -> Recorded 200 new commits
[GitCheckoutListener] -> The latest commit 'e7ce93ef07cdaff3ae372dfed57c93ccf3ebbd84' is a merge commit
[GitCheckoutListener] -> Git commit decorator successfully obtained 'hudson.plugins.git.browser.BitbucketWeb@72f603c0' to render commit links
Finished: SUCCESS
Failed log
Started by user User name
Running as SYSTEM
[EnvInject] - Loading node environment variables.
[EnvInject] - Preparing an environment for the build.
[EnvInject] - Keeping Jenkins system variables.
[EnvInject] - Keeping Jenkins build variables.
[EnvInject] - Injecting as environment variables the properties content
GIT_TRACE=1
[EnvInject] - Variables injected successfully.
[EnvInject] - Injecting contributions.
Building remotely on company-admdb (applprod22_linux_db linux appltest22_linux_db) in workspace /var/lib/jenkins/workspace/cceadm appltest22/xcomp checkout
[EnvInject] - Unset unresolved 'BASH_FUNC_which%%' variable.
The recommended git tool is: NONE
using credential tokentokenTOKENTOKEN
Cloning the remote Git repository
Cloning repository https://bitbucket.org/company/reponame.git
> git init /var/lib/jenkins/workspace/cceadm appltest22/xcomp checkout # timeout=10
Fetching upstream changes from https://bitbucket.org/company/reponame.git
> git --version # timeout=10
> git --version # 'git version 2.31.1'
using GIT_ASKPASS to set credentials token TOKENTOKEN
> git fetch --tags --force --progress -- https://bitbucket.org/company/reponame.git +refs/heads/*:refs/remotes/origin/* # timeout=10
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "git fetch --tags --force --progress -- https://bitbucket.org/company/reponame.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout:
stderr: 18:54:05.328181 git.c:447 trace: built-in: git fetch --tags --force --progress -- https://bitbucket.org/company/reponame.git '+refs/heads/*:refs/remotes/origin/*'
18:54:05.328405 run-command.c:667 trace: run_command: GIT_DIR=.git git remote-https https://bitbucket.org/company/reponame.git https://x-token-auth:*******************************************************************************************@bitbucket.org/company/reponame.git
18:54:05.330301 git.c:733 trace: exec: git-remote-https https://bitbucket.org/company/reponame.git https://x-token-auth:*******************************************************************************************@bitbucket.org/company/reponame.git
18:54:05.330335 run-command.c:667 trace: run_command: git-remote-https https://bitbucket.org/company/reponame.git https://x-token-auth:*******************************************************************************************@bitbucket.org/company/reponame.git
remote: The requested repository either does not exist or you do not have access. If you believe this repository exists and you have access, make sure you're authenticated.
fatal: unable to access 'https://bitbucket.org/company/reponame.git/': The requested URL returned error: 403
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2842)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:635)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:871)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:377)
at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to company-admdb
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1787)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
at hudson.remoting.Channel.call(Channel.java:1003)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:153)
at jdk.internal.reflect.GeneratedMethodAccessor2090.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:138)
at jdk.proxy123/jdk.proxy123.$Proxy256.execute(Unknown Source)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1222)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1305)
at hudson.scm.SCM.checkout(SCM.java:540)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1248)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
at hudson.model.Run.execute(Run.java:1895)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)
ERROR: Error cloning remote repo 'origin'
Finished: FAILURE
jenkins setup
Jenkins: 2.440.1
OS: Linux - 3.10.0-1160.31.1.el7.x86_64
Java: 17.0.10 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
Office-365-Connector:4.21.0
active-directory:2.35
ansicolor:1.0.4
ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
authentication-tokens:1.53.v1c90fd9191a_b_
bitbucket-build-status-notifier:1.4.2
bitbucket-filter-project-trait:1.0
bitbucket-oauth:0.13
bitbucket-push-and-pull-request:3.0.2
blueocean:1.27.11
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.11
blueocean-commons:1.27.11
blueocean-config:1.27.11
blueocean-core-js:1.27.11
blueocean-dashboard:1.27.11
blueocean-display-url:2.4.2
blueocean-events:1.27.11
blueocean-git-pipeline:1.27.11
blueocean-github-pipeline:1.27.11
blueocean-i18n:1.27.11
blueocean-jira:1.27.11
blueocean-jwt:1.27.11
blueocean-personalization:1.27.11
blueocean-pipeline-api-impl:1.27.11
blueocean-pipeline-editor:1.27.11
blueocean-pipeline-scm-api:1.27.11
blueocean-rest:1.27.11
blueocean-rest-impl:1.27.11
blueocean-web:1.27.11
bootstrap5-api:5.3.2-4
bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9
branch-api:2.1152.v6f101e97dd77
build-monitor-plugin:1.14-860.vd06ef2568b_3f
build-timeout:1.32
caffeine-api:3.1.8-133.v17b_1ff2e0599
calendar-view:0.3.3
checks-api:2.0.2
cloudbees-bitbucket-branch-source:877.vb_b_d5243f6794
cloudbees-disk-usage-simple:203.v3f46a_7462b_1a_
cloudbees-folder:6.901.vb_4c7a_da_75da_3
command-launcher:107.v773860566e2e
commons-httpclient3-api:3.1-3
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.11.0-95.v22a_d30ee5d36
configuration-as-code:1775.v810dc950b_514
credentials:1319.v7eb_51b_3a_c97b_
credentials-binding:657.v2b_19db_7d6e6d
dark-theme:439.vdef09f81f85e
dashboard-view:2.508.va_74654f026d1
data-tables-api:1.13.8-4
display-url-api:2.200.vb_9327d658781
docker-commons:439.va_3cb_0a_6a_fb_29
docker-workflow:572.v950f58993843
durable-task:550.v0930093c4b_a_6
echarts-api:5.4.3-4
email-ext:2.104
embeddable-build-status:467.v4a_954796e45d
envinject:2.908.v66a_774b_31d93
envinject-api:1.199.v3ce31253ed13
external-monitor-job:215.v2e88e894db_f8
extra-columns:1.26
favorite:2.208.v91d65b_7792a_c
folder-properties:1.2.1
font-awesome-api:6.5.1-3
forensics-api:2.4.0
generic-webhook-trigger:2.0.0
git:5.2.1
git-client:4.6.0
git-forensics:2.1.0
git-server:114.v068a_c7cc2574
github:1.38.0
github-api:1.318-461.v7a_c09c9fa_d63
github-branch-source:1772.va_69eda_d018d4
gradle:2.10
gravatar:2.2
gson-api:2.10.1-15.v0d99f670e0a_7
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
htmlpublisher:1.32
instance-identity:185.v303dc7c645f9
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.16.1-373.ve709c6871598
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javadoc:243.vb_b_503b_b_45537
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jenkins-design-language:1.27.11
jersey2-api:2.41-133.va_03323b_a_1396
jfrog:1.5.0
jira:3.12
jjwt-api:0.11.5-77.v646c772fddb_0
jnr-posix-api:3.1.19-1
job-dsl:1.87
jobConfigHistory:1229.v3039470161a_d
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery:1.12.4-1
jquery3-api:3.7.1-2
jsch:0.2.16-86.v42e010d9484b_
json-api:20240205-27.va_007549e895c
json-path-api:2.9.0-33.v2527142f2e1d
junit:1259.v65ffcef24a_88
ldap:711.vb_d1a_491714dc
locale:431.v3435fa_8f8445
lockable-resources:1243.v346d600eea_24
log-file-filter:116.v881e1b_75ed59
mailer:463.vedf8358e006b_
mapdb-api:1.0.9-28.vf251ce40855d
matrix-auth:3.2.1
matrix-project:822.824.v14451b_c0fd42
mercurial:1260.vdfb_723cdcc81
metrics:4.2.21-449.v6960d7c54c69
mina-sshd-api-common:2.12.0-90.v9f7fb_9fa_3d3b_
mina-sshd-api-core:2.12.0-90.v9f7fb_9fa_3d3b_
multiple-scms:0.8
multiselect-parameter:1.7
naginator:1.436.vb_e769dcb_cdf6
next-executions:302.vf833440a_3fd6
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
openedge:0.9
opentelemetry:3.1111.vc2733c03b_db_1
pam-auth:1.10
parameterized-scheduler:262.v00f3d90585cc
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-graph-view:224.vef989d141666
pipeline-groovy-lib:704.vc58b_8890a_384
pipeline-input-step:491.vb_07d21da_1a_fb_
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2175.v76a_fff0a_2618
pipeline-model-definition:2.2175.v76a_fff0a_2618
pipeline-model-extensions:2.2175.v76a_fff0a_2618
pipeline-rest-api:2.34
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2175.v76a_fff0a_2618
pipeline-stage-view:2.34
pipeline-utility-steps:2.16.2
plain-credentials:143.v1b_df8b_d3b_e48
plugin-usage-plugin:4.2
plugin-util-api:4.1.0
prism-api:1.29.0-13
prometheus:2.5.1
publish-over:0.22
pubsub-light:1.18
resource-disposer:0.23
scm-api:683.vb_16722fb_b_80b_
script-security:1326.vdb_c154de8669
show-build-parameters:1.0
sidebar-link:2.4.1
snakeyaml-api:2.2-111.vc6598e30cc65
sonar:2.17.2
sonar-quality-gates:1.3.1
sse-gateway:1.26
ssh-credentials:308.ve4497b_ccd8f4
ssh-slaves:2.948.vb_8050d697fec
sshd:3.322.v159e91f6a_550
structs:337.v1b_04ea_4df7c8
subversion:2.17.3
theme-manager:215.vc1ff18d67920
timestamper:1.26
token-macro:400.v35420b_922dcb_
trilead-api:2.133.vfb_8a_7b_9c5dd1
variant:60.v7290fc0eb_b_cd
view-job-filters:369.ve0513a_a_f5524
workflow-aggregator:596.v8c21c963d92d
workflow-api:1291.v51fd2a_625da_7
workflow-basic-steps:1042.ve7b_140c4a_e0c
workflow-cps:3880.vb_ef4b_5cfd270
workflow-durable-task-step:1331.vc8c2fed35334
workflow-job:1400.v7fd111b_ec82f
workflow-multibranch:773.vc4fe1378f1d5
workflow-scm-step:415.v434365564324
workflow-step-api:657.v03b_e8115821b_
workflow-support:865.v43e78cc44e0d
ws-cleanup:0.45
