All jobs hanging on Git operations on controller

Jenkins setup

Jenkins: 2.555.3

OS: Linux - 6.1.182
Java: 21.0.11 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)

amazon-ecs:1.49
analysis-model-api:14.13.0-998.v6dec20409ce6
ant:520.vd082ecfb_16a_9
antisamy-markup-formatter:173.v680e3a_b_69ff3
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
apache-httpcomponents-client-5-api:5.6.1-195.v65ffe15189a_d
artifact-manager-s3:973.v6a_51253e896b_
artifactory:4.0.8
asm-api:9.10.1-216.va_9256d3b_844b_
atlassian-bitbucket-server-integration:6.0.0
authentication-tokens:1.144.v5ff4a_5ec5c33
aws-credentials:262.v0efb_28f71f94
aws-global-configuration:159.v882a_ecd03fe6
aws-java-sdk:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-api-gateway:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-autoscaling:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudformation:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudfront:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudwatch:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-codebuild:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-codedeploy:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ec2:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ecr:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ecs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-efs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-elasticbeanstalk:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-elasticloadbalancingv2:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-iam:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-kinesis:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-lambda:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-logs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-minimal:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-organizations:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-secretsmanager:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-sns:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-sqs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ssm:1.12.780-480.v4a_0819121a_9e
aws-java-sdk2-autoscaling:2.42.33-70.vd69c0763fa_60
aws-java-sdk2-cloudformation:2.42.33-70.vd69c0763fa_60
aws-java-sdk2-core:2.42.33-70.vd69c0763fa_60
aws-java-sdk2-ec2:2.42.33-70.vd69c0763fa_60
aws-java-sdk2-s3:2.42.33-70.vd69c0763fa_60
aws-java-sdk2-secretsmanager:2.42.33-70.vd69c0763fa_60
aws-secrets-manager-credentials-provider:2.222.v376939a_9ffb_b
aws-secrets-manager-secret-source:2.87.v49cc86c50c3b_
basic-branch-build-strategies:317.v85b_331d6cc42
bitbucket:307.va_9f30d0b_8f0d
blueocean:1.27.25
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.25
blueocean-commons:1.27.25
blueocean-config:1.27.25
blueocean-core-js:1.27.25
blueocean-dashboard:1.27.25
blueocean-display-url:2.4.4
blueocean-events:1.27.25
blueocean-git-pipeline:1.27.25
blueocean-github-pipeline:1.27.25
blueocean-i18n:1.27.25
blueocean-jira:1.27.25
blueocean-jwt:1.27.25
blueocean-personalization:1.27.25
blueocean-pipeline-api-impl:1.27.25
blueocean-pipeline-editor:1.27.25
blueocean-pipeline-scm-api:1.27.25
blueocean-rest:1.27.25
blueocean-rest-impl:1.27.25
blueocean-web:1.27.25
bootstrap5-api:5.3.8-1048.va_c299057e35c
bouncycastle-api:2.30.1.84-291.v9f17b_21896e2
branch-api:2.1280.v0d4e5b_b_460ef
build-name-setter:2.5.1
build-timeout:1.40
caffeine-api:3.2.3-194.v31a_b_f7a_b_5a_81
checks-api:415.vf022234a_931d
cloudbees-bitbucket-branch-source:937.3.2
cloudbees-folder:6.1100.ve9eed61d16c4
command-launcher:134.v025a_5fcf9dea_
commons-collections4-api:4.5.0-8.va_d5448ef9011
commons-compress-api:1.28.0-82.v4df6c060eb_66
commons-lang3-api:3.20.0-109.ve43756e2d2b_4
commons-text-api:1.15.0-218.va_61573470393
config-file-provider:1013.v73c323e52b_1f
configuration-as-code:2089.v970a_0b_a_8cc6d
copyartifact:795.ve8e151429b_27
credentials:1502.v5c95e620ddfe
credentials-binding:725.ve52b_2328a_fde
custom-tools-plugin:0.8
data-tables-api:2.3.8-1570.v1cb_1cd2a_0fb_c
display-url-api:2.217.va_6b_de84cc74b_
docker-commons:472.vee120e23d3a_c
docker-workflow:634.vedc7242b_eda_7
dotnet-sdk:1.4.0
dtkit-api:3.0.3
durable-task:671.v340ff7959010
ec2:2053.v86eb_364849fc
ec2-fleet:4.2.3.540.va_6eedb_7b_c112
echarts-api:6.0.0-1287.vfd24c22a_3d00
eddsa-api:0.3.0.1-29.v67e9a_1c969b_b_
email-ext:2038.v7b_8817a_499d9
emailext-template:249.v81d6ea_f33f2e
embeddable-build-status:637.vd878e68178f8
extended-choice-parameter:388.ve7b_d0b_920e10
extended-read-permission:68.vd270568a_7520
favorite:2.263.v941d21defef7
font-awesome-api:7.2.0-990.vf220b_2a_496f9
forensics-api:4.1891.v5e60f3377506
generic-webhook-trigger:2.4.2
git:5.10.1
git-client:6.6.1
github:1.47.0
github-api:1.330-492.v3941a_032db_2a_
github-branch-source:1967.1970.vd86979736546
gradle:2.19.1244.v1f9866817fec
gson-api:2.14.0-201.v8eefe5515533
handy-uri-templates-2-api:2.1.8-38.vcea_5d521d5f3
htmlpublisher:427.1
http_request:1.25
instance-identity:203.v15e81a_1b_7a_38
ionicons-api:94.vcc3065403257
jackson-annotations2-api:2.22-19.v10a_a_582ea_26e
jackson2-api:2.21.2-436.v29efdb_7418ff
jackson3-api:3.2.0-83.vfb_04cee1ea_0f
jakarta-activation-api:2.1.4-1
jakarta-mail-api:2.1.5-1
jakarta-xml-bind-api:4.0.9-19.v2b_a_5b_44d9a_1c
javadoc:354.vee1a_660b_4990
javax-activation-api:1.2.0-8
javax-mail-api:1.6.2-11
jaxb:2.3.9-143.v5979df3304e6
jdk-tool:83.v417146707a_3d
jenkins-design-language:1.27.25
jersey2-api:2.48-180.ve47b_264f849b_
jira:3.21
jjwt-api:0.13.0-141.vd58b_a_9592b_6c
job-dsl:3654.vdf58f53e2d15
jobcacher:842.v74b_a_026267b_6
joda-time-api:2.14.2-193.v422b_efce56e0
jquery3-api:3.7.1-687.v68d468e40b_30
jsch:0.2.16-95.v3eecb_55fa_b_78
json-api:20260522-217.v0b_18b_8cd4672
json-path-api:3.0.0-218.vcd4dd1355de2
jsoup:1.22.2-95.vc5d00f1eb_42d
junit:1413.v736fa_5b_61d80
mailer:534.v1b_36f5864073
matrix-auth:3.2.10
matrix-project:870.v9db_fcfc2f45b_
maven-plugin:3.27
mercurial:1323.ve69d2a_db_8a_b_d
metrics:4.2.37-494.v06f9a_939d33a_
mina-sshd-api-common:2.17.1-187.v0341274c2905
mina-sshd-api-core:2.17.1-187.v0341274c2905
mina-sshd-api-scp:2.17.1-187.v0341274c2905
monitoring:2.8.0
msbuild:1.42
mstest:1.0.5
node-iterator-api:72.vc90e81737df1
nodejs:1.6.6
okhttp-api:5.3.2-200.vedb_720a_cf1f8
oss-symbols-api:465.v25d9fdc88c26
pam-auth:1.12
parameterized-trigger:893.va_383a_9b_a_4c11
pipeline-aws:1.45
pipeline-build-step:584.vdb_a_2cc3a_d07a_
pipeline-graph-analysis:254.v0f63a_a_447dca_
pipeline-graph-view:949.v1cf0f01ca_cfa_
pipeline-groovy-lib:798.v5cc688825312
pipeline-input-step:551.vdff487c5998c
pipeline-milestone-step:152.v6e22b_8cfc66c
pipeline-model-api:2.2291.v2934911987b_6
pipeline-model-definition:2.2291.v2934911987b_6
pipeline-model-extensions:2.2291.v2934911987b_6
pipeline-rest-api:2.41
pipeline-stage-step:345.va_96187909426
pipeline-stage-tags-metadata:2.2291.v2934911987b_6
pipeline-stage-view:2.41
pipeline-timeline:1.0.3
pipeline-utility-steps:3.810.va_7672d206740
plain-credentials:199.v9f8e1f741799
plugin-util-api:7.1341.v039f146993d9
powershell:185.v7a_026da_c54ee
prism-api:1.30.0-741.v034eb_0b_0a_a_fa_
pubsub-light:1.19
resource-disposer:0.25
role-strategy:867.vd09254229f9b_
run-condition:276.v97298f3a_cd51
saml:4.600.v069e2e79eeb_f
scm-api:728.vc30dcf7a_0df5
script-security:1402.1405.vc96e74964250
snakeyaml-api:2.5-149.v72471e9c6371
snakeyaml-engine-api:3.0.1-5.vd98ea_ff3b_92e
sonar:2.18.3
sse-gateway:1.29
ssh-agent:396.vcc7d84e622ec
ssh-credentials:372.va_250881b_08cd
ssh-slaves:3.1097.v868116049892
sshd:3.384.vc89b_5e138cf9
structs:362.va_b_695ef4fdf9
timestamper:1.30
token-macro:477.vd4f0dc3cb_cf1
trilead-api:2.284.v1974ea_324382
variant:70.va_d9f17f859e0
woodstox-core-api:7.2.1-6.v3718a_a_11f5c4
workflow-aggregator:608.v67378e9d3db_1
workflow-api:1413.v2ff1a_5e720fa_
workflow-basic-steps:1098.v808b_fd7f8cf4
workflow-cps:4331.4333.v50a_b_076c5199
workflow-durable-task-step:1479.v56e587f413a_7
workflow-job:1571.1580.v18e46842c125
workflow-multibranch:821.vc3b_4ea_780798
workflow-scm-step:466.va_d69e602552b_
workflow-step-api:724.v538c2362b_dfb_
workflow-support:1015.v785e5a_b_b_8b_22
ws-cleanup:0.49
xunit:3.1.6

We have had an issue twice now where all jobs starting will get stuck on the controller during the initial Git operations to get the shared libraries and Jenkinsfile before the jobs is assigned to a node. The only resolution we have found for the issue is restarting the controller. Has anyone else experienced an issue like this? It does not seem to correlate to any instance of high CPU utilization, and I was not seeing any thread blocking. I am not sure if this is an issue in Jenkins core, the Git plugin, Git itself, or something else to know where best to open an issue.

You may be affected by the overly aggressive git hook configuration locking that is described in:

Could you test the incremental build and see if it resolves your issue?

It seems that the incremental build doesn’t exist in Artifactory

Hi

You can build with the patch applied and test it.

I merged the controller branch into that pull request and it generated a new incremental build. That incremental build will remain in the cache for up to 3 months, then it will be removed.