Jenkins agent git clone issues

Hi all, I set up a Jenkins CI with a controller controller on a Windows Server (see Jenkins setup below) and 3 agent on Mac mini running Sonoma. Mac minis too are using Java 21. Every machine is running Jenkins or its agents directly: no container nor other virtualization is used. I’m using some declarative pipelines I wrote myself.

Everything was running perfectly for 8 months, today I discovered that 2 nodes out of three had issue performing clone on our repo.
The error message is the following:

14:22:41 + git clone git@bitbucket.org:xxxyyyzzz/ios.git .
14:22:41 Cloning into ‘.’…
14:22:41 No user exists for uid 501 14:22:41 fatal: Could not read from remote repository.

I got this error while performing a “simple” clone on out repo, both using a shell script or using Jenkins’ own git code inside declarative pipeline. The same pipeline and shell script al well don’t have any issue on the only machine which is still working.

I tried performing some debug, but I could not manage to find anything in particular but agent uptime. The only one out of three which was working was recently restarted (6th of August) while the others two were more or less three months old. Anyway, I restarted the agents and everything started working again as usual. I really like to understand what’s happened!

Thanks in advance for any tips

Regards

Marco

Jenkins setup:
Jenkins: 2.452.2
OS: Windows Server 2019 - 10.0
Java: 21.0.2 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)

ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
artifactdeployer:1.3
authentication-tokens:1.53.v1c90fd9191a_b_
authorize-project:1.7.1
azure-credentials:293.vb_d506148f506
azure-keyvault:214.vf667264ea_ccd
azure-sdk:132.v62b_48eb_6f32f
bitbucket:223.vd12f2bca5430
blueocean:1.27.8
blueocean-bitbucket-pipeline:1.27.8
blueocean-commons:1.27.11
blueocean-config:1.27.8
blueocean-core-js:1.27.8
blueocean-dashboard:1.27.8
blueocean-display-url:2.4.2
blueocean-events:1.27.8
blueocean-git-pipeline:1.27.8
blueocean-github-pipeline:1.27.8
blueocean-i18n:1.27.8
blueocean-jwt:1.27.8
blueocean-personalization:1.27.8
blueocean-pipeline-api-impl:1.27.8
blueocean-pipeline-editor:1.27.8
blueocean-pipeline-scm-api:1.27.8
blueocean-rest:1.27.11
blueocean-rest-impl:1.27.8
blueocean-web:1.27.8
bootstrap5-api:5.3.2-3
bouncycastle-api:2.30.1.78.1-233.vfdcdeb_0a_08a_a_
branch-api:2.1128.v717130d4f816
build-timeout:1.31
build-user-vars-plugin:1.9
built-on-column:1.4
bulk-builder:1.5
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.0.2
cloudbees-bitbucket-branch-source:887.va_d359b_3d2d8d
cloudbees-folder:6.848.ve3b_fd7839a_81
command-launcher:107.v773860566e2e
commons-httpclient3-api:3.1-3
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.10.0-78.v3e7b_ea_d5a_fe1
coverage:1.10.0
credentials:1344.v5a_3f65a_1e173
credentials-binding:657.v2b_19db_7d6e6d
dark-theme:439.vdef09f81f85e
data-tables-api:1.13.8-2
display-url-api:2.200.vb_9327d658781
dtkit-api:3.0.2
durable-task:523.va_a_22cf15d5e0
echarts-api:5.4.3-2
email-ext:2.102
envinject:2.908.v66a_774b_31d93
envinject-api:1.199.v3ce31253ed13
environment-script:1.2.6
external-monitor-job:215.v2e88e894db_f8
favorite:2.4.3
font-awesome-api:6.4.2-1
forensics-api:2.3.0
git:5.2.1
git-client:4.5.0
github:1.37.3.1
github-api:1.316-451.v15738eef3414
github-branch-source:1741.va_3028eb_9fd21
github-checks:554.vb_ee03a_000f65
gradle:2.8.2
gson-api:2.10.1-15.v0d99f670e0a_7
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
htmlpublisher:1.33
hudson-wsclean-plugin:1.0.8
instance-identity:185.v303dc7c645f9
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.16.1-373.ve709c6871598
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javadoc:243.vb_b_503b_b_45537
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jenkins-design-language:1.27.8
jjwt-api:0.11.5-77.v646c772fddb_0
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery3-api:3.7.1-1
jsch:0.2.16-86.v42e010d9484b_
json-path-api:2.9.0-33.v2527142f2e1d
junit:1265.v65b_14fa_f12f0
ldap:701.vf8619de9160a_
locale:314.v22ce953dfe9e
lockable-resources:1185.v0c528656ce04
mailer:472.vf7c289a_4b_420
matrix-auth:3.2.2
matrix-project:822.824.v14451b_c0fd42
mercurial:1260.vdfb_723cdcc81
mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_
mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_
msbuild:1.30
mstest:1.0.3
mstestrunner:1.5.0
multibranch-action-triggers:1.8.6
multibranch-build-strategy-extension:1.0.10
nodelabelparameter:1.12.0
okhttp-api:4.11.0-157.v6852a_a_fa_ec11
pam-auth:1.10
parameter-separator:166.vd0120849b_386
pipeline-build-step:505.v5f0844d8d126
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:689.veec561a_dee13
pipeline-input-step:477.v339683a_8d55e
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2144.v077a_d1928a_40
pipeline-model-definition:2.2144.v077a_d1928a_40
pipeline-model-extensions:2.2144.v077a_d1928a_40
pipeline-rest-api:2.33
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2144.v077a_d1928a_40
pipeline-stage-view:2.33
plain-credentials:183.va_de8f1dd5a_2b_
plugin-util-api:3.8.0
powershell:2.1
prism-api:1.29.0-10
proc-cleaner-plugin:2.4
pubsub-light:1.18
resource-disposer:0.23
role-strategy:689.v731678c3e0eb_
scm-api:676.v886669a_199a_a_
script-security:1336.vf33a_a_9863911
short-workspace-path:0.3
snakeyaml-api:2.2-111.vc6598e30cc65
sonar:2.16
sse-gateway:1.26
ssh-credentials:308.ve4497b_ccd8f4
ssh-slaves:2.916.vd17b_43357ce4
sshd:3.322.v159e91f6a_550
structs:338.v848422169819
theme-manager:215.vc1ff18d67920
thinBackup:1.18
timestamper:1.26
token-macro:384.vf35b_f26814ec
trilead-api:2.142.v748523a_76693
uno-choice:2.7.2
variant:60.v7290fc0eb_b_cd
workflow-aggregator:596.v8c21c963d92d
workflow-api:1283.v99c10937efcb_
workflow-basic-steps:1042.ve7b_140c4a_e0c
workflow-cps:3802.vd42b_fcf00b_a_c
workflow-durable-task-step:1289.v4d3e7b_01546b_
workflow-job:1348.v32a_a_f150910e
workflow-multibranch:756.v891d88f2cd46
workflow-scm-step:415.v434365564324
workflow-step-api:657.v03b_e8115821b_
workflow-support:865.v43e78cc44e0d
ws-cleanup:0.45
xunit:3.1.3

looks like there is no user with uid 501 . YOu need to have a generic user created and set .gitconfig to perform cloning or you have to your user credentials set on Jenkins server and try cloning as yourself.

All of my 3 nodes are configured with keys exchange to clone repo on bitbucket.
They work flawlessly normally, that particular day they 2 of them started giving me that error while trying to clone.
What is really weird is that when I got this error I tried to log on that nodes and tried git cloning on a shell by myself and I had no problem! Whine on the same node, my pipelines failed with that error when trying to clone from the same repo launching git clone via shell script.
The only solution I found to solve the issue was to restart the Jenkins nodes, not even the machines they run on, just the nodes!
How is it possible?