Scan Multibranch Pipeline Git Packed-Refs issue

Hi all. We’re running into an issue with an issue scanning the main repo that’s used for several different pipeline jobs. Right now whenever a job scans this repo to look for branches to build, I get the following error at the end of the scanning log:

> git.exe fetch --tags --force --progress --prune -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10
ERROR: [Mon Oct 23 14:28:56 PDT 2023] Could not fetch branches from source 28812075-85b0-489f-954e-ce1d47119ee0
[Mon Oct 23 14:28:56 PDT 2023] Finished branch indexing. Indexing took 4.3 sec
FATAL: Failed to recompute children of <redacted>
hudson.plugins.git.GitException: Command "git.exe fetch --tags --force --progress --prune -- origin +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: fatal: unterminated line in .git/packed-refs: 

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2842)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:635)
	at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:605)
	at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:591)
	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:397)
	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:351)
	at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:591)
	at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:372)
	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:282)
	at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:654)
	at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:269)
	at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:167)
	at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1057)
	at hudson.model.ResourceController.execute(ResourceController.java:101)
	at hudson.model.Executor.run(Executor.java:442)
Finished: FAILURE

Looking at this error online, the consensus seems to be to just delete the .git/packed-refs file. The problem is I don’t know where that file is in this case, since its not even gotten to the part where its building anything yet. I’ve not been able to find it in the Jenkins server. And since files in .git aren’t tracked (and don’t look like they can be from what I’ve tried), I can’t update the repo to have this file deleted.
Any ideas?

Jenkins setup:
Jenkins: 2.414.2
OS: Windows 10 - 10.0
Java: 17.0.8 - Oracle Corporation (Java HotSpot™ 64-Bit Server VM)

ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
authentication-tokens:1.53.v1c90fd9191a_b_
bitbucket:223.vd12f2bca5430
bitbucket-push-and-pull-request:2.8.4
bootstrap5-api:5.3.2-1
bouncycastle-api:2.29
branch-api:2.1128.v717130d4f816
build-name-setter:2.3.0
build-timeout:1.31
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.0.2
cloudbees-bitbucket-branch-source:845.v27a_d5823911b_
cloudbees-folder:6.848.ve3b_fd7839a_81
command-launcher:107.v773860566e2e
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.10.0-78.v3e7b_ea_d5a_fe1
credentials:1271.v54b_1c2c6388a_
credentials-binding:636.v55f1275c7b_27
dashboard-view:2.495.v07e81500c3f2
display-url-api:2.200.vb_9327d658781
durable-task:523.va_a_22cf15d5e0
echarts-api:5.4.0-6
email-ext:2.102
font-awesome-api:6.4.2-1
git:5.2.0
git-client:4.5.0
github:1.37.3
github-api:1.316-451.v15738eef3414
github-branch-source:1741.va_3028eb_9fd21
gradle:2.8.2
handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
instance-identity:173.va_37c494ec4e5
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.15.2-350.v0c2f3f8fc595
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.8-1
jdk-tool:73.vddf737284550
jjwt-api:0.11.5-77.v646c772fddb_0
job-dsl:1.85
jquery3-api:3.7.1-1
jsch:0.2.8-65.v052c39de79b_2
junit:1240.vf9529b_881428
ldap:701.vf8619de9160a_
mailer:463.vedf8358e006b_
matrix-auth:3.2.1
matrix-project:808.v5a_b_5f56d6966
mercurial:1260.vdfb_723cdcc81
mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_
mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_
okhttp-api:4.11.0-157.v6852a_a_fa_ec11
pam-auth:1.10
pipeline-build-step:505.v5f0844d8d126
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:689.veec561a_dee13
pipeline-input-step:477.v339683a_8d55e
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2144.v077a_d1928a_40
pipeline-model-definition:2.2144.v077a_d1928a_40
pipeline-model-extensions:2.2144.v077a_d1928a_40
pipeline-rest-api:2.33
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2144.v077a_d1928a_40
pipeline-stage-view:2.33
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:3.4.0
resource-disposer:0.23
scm-api:676.v886669a_199a_a_
script-security:1275.v23895f409fb_d
snakeyaml-api:2.2-111.vc6598e30cc65
ssh-agent:333.v878b_53c89511
ssh-credentials:308.ve4497b_ccd8f4
ssh-slaves:2.916.vd17b_43357ce4
sshd:3.312.v1c601b_c83b_0e
structs:325.vcb_307d2a_2782
timestamper:1.26
token-macro:384.vf35b_f26814ec
trilead-api:2.84.v72119de229b_7
variant:60.v7290fc0eb_b_cd
workflow-aggregator:596.v8c21c963d92d
workflow-api:1283.v99c10937efcb_
workflow-basic-steps:1042.ve7b_140c4a_e0c
workflow-cps:3793.v65dec41c3a_c3
workflow-durable-task-step:1289.v4d3e7b_01546b_
workflow-job:1348.v32a_a_f150910e
workflow-multibranch:756.v891d88f2cd46
workflow-scm-step:415.v434365564324
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:865.v43e78cc44e0d
ws-cleanup:0.45

How about:

cd $JENKINS_HOME && find . -type f -name packed-refs -print

That should find all files in the Jenkins home directory that have the name packed-refs. Then you can decide which of those files should be deleted.

Thanks for the reply. I was able to track down the packed-refs file in the newest folder under $JENKINS_HOME\caches. I’ve had to manually clear out the cache before, but its been a while so I had forgotten about it. The most recent cache had that packed-refs file, and I also had to delete all the folders under $JENKINS_HOME\caches\git-<uuid>\.git\refs. With those files/folders deleted, the pipeline scan rebuilt its cache and worked fine.