Thousands of missing folders in Jenkins home directory

Using Jenkins
, ,
1

Hello guys,

We have Jenkins deployed in Kubernetes.
When we upgrade the server it takes too long too boot up and also sometimes restarts during the startup time.
Most of the logs something like this:

2023-10-27 12:08:51.054+0000 [id=29]	INFO	c.c.h.p.folder.AbstractFolder$3#call: Loading job Engineering/Markup Editor/Publish NPM Package/bugfix%2FBBESS-1277 (4048.3%)
2023-10-27 12:08:54.333+0000 [id=29]	WARNING	c.c.h.p.folder.AbstractFolder#loadChildren: could not find file /var/jenkins_home/jobs/Engineering/jobs/Markup Editor/jobs/Publish NPM Package/branches/PR-74/config.xml
2023-10-27 12:09:02.491+0000 [id=29]	WARNING	c.c.h.p.folder.AbstractFolder#loadChildren: could not find file /var/jenkins_home/jobs/Engineering/jobs/Markup Editor/jobs/Publish NPM Package/branches/PR-71/config.xml
2023-10-27 12:09:06.061+0000 [id=29]	INFO	c.c.h.p.folder.AbstractFolder$3#call: Loading job Engineering/RealTime-Collaboration-Service/RTCS-CD/feature%2FBBESS-1042-provisioning-infrastructure (4165.9%)
2023-10-27 12:09:06.069+0000 [id=29]	WARNING	c.c.h.p.folder.AbstractFolder#loadChildren: could not find file /var/jenkins_home/jobs/Engineering/jobs/RealTime-Collaboration-Service/jobs/RTCS-CD/branches/feature-test.f0qhbc/config.xml
2023-10-27 12:09:06.774+0000 [id=29]	WARNING	c.c.h.p.folder.AbstractFolder#loadChildren: could not find file /var/jenkins_home/jobs/Engineering/jobs/RealTime-Collaboration-Service/jobs/RTCS-CI/branches/feature-B.uqlrr0u07qv2.nters-403/config.xml
2023-10-27 12:09:08.071+0000 [id=29]	WARNING	c.c.h.p.folder.AbstractFolder#loadChildren: could not find file /var/jenkins_home/jobs/Engineering/jobs/RealTime-Collaboration-Service/jobs/RTCS-CI/branches/feature-test.f0qhbc/config.xml
2023-10-27 12:09:13.989+0000 [id=29]	WARNING	c.c.h.p.folder.AbstractFolder#loadChildren: could not find file /var/jenkins_home/jobs/Engineering/jobs/editor/jobs/integration/jobs/mera-react/branches/PR-10/config.xml
2023-10-27 12:09:14.572+0000 [id=31]	WARNING	c.c.h.p.folder.AbstractFolder#loadChildren: could not find file /var/jenkins_home/jobs/EngineeringConsultants/jobs/SubscriptionLicensing/jobs/TrialService.ci/branches/feature-BIIT-1369b.pes4mr/config.xml
2023-10-27 12:09:14.642+0000 [id=29]	WARNING	c.c.h.p.folder.AbstractFolder#loadChildren: could not find file /var/jenkins_home/jobs/Engineering/jobs/editor/jobs/integration/jobs/mera-react/branches/PR-9/config.xml
2023-10-27 12:09:15.258+0000 [id=29]	WARNING	c.c.h.p.folder.AbstractFolder#loadChildren: could not find file /var/jenkins_home/jobs/Engineering/jobs/editor/jobs/integration/jobs/mera-react/branches/PR-5/config.xml
2023-10-27 12:09:15.687+0000 [id=29]	WARNING	c.c.h.p.folder.AbstractFolder#loadChildren: could not find file /var/jenkins_home/jobs/Engineering/jobs/editor/jobs/integration/jobs/mera-react/branches/feat-add-image-markups.0nqb7v/config.xml
2023-10-27 12:09:16.731+0000 [id=29]	WARNING	c.c.h.p.folder.AbstractFolder#loadChildren: could not find file /var/jenkins_home/jobs/Engineering/jobs/editor/jobs/integration/jobs/mera-react/branches/PR-7/config.xml
2023-10-27 12:09:17.498+0000 [id=29]	WARNING	c.c.h.p.folder.AbstractFolder#loadChildren: could not find file /var/jenkins_home/jobs/Engineering/jobs/editor/jobs/integration/jobs/mera-react/branches/refactor-tak.1g0hl0.er-editor-ui/config.xml
2023-10-27 12:09:17.558+0000 [id=31]	WARNING	c.c.h.p.folder.AbstractFolder#loadChildren: could not find file /var/jenkins_home/jobs/EngineeringConsultants/jobs/SubscriptionLicensing/jobs/TrialService.ci/branches/fix-trial-we.ge3tdo.e-user-login/config.xml
2023-10-27 12:09:17.635+0000 [id=29]	WARNING	c.c.h.p.folder.AbstractFolder#loadChildren: could not find file /var/jenkins_home/jobs/Engineering/jobs/editor/jobs/integration/jobs/mera-react/branches/feat-page-rotation.isb2th/config.xml
2023-10-27 12:09:17.707+0000 [id=29]	WARNING	c.c.h.p.folder.AbstractFolder#loadChildren: could not find file /var/jenkins_home/jobs/Engineering/jobs/editor/jobs/integration/jobs/mera-react/branches/bugfix-BBCLD.78ugjr.959-Extended/config.xml
2023-10-27 12:09:18.733+0000 [id=29]	WARNING	c.c.h.p.folder.AbstractFolder#loadChildren: could not find file /var/jenkins_home/jobs/Engineering/jobs/editor/jobs/integration/jobs/mera-react/branches/PR-6/config.xml
2023-10-27 12:09:20.531+0000 [id=29]	WARNING	c.c.h.p.folder.AbstractFolder#loadChildren: could not find file /var/jenkins_home/jobs/Engineering/jobs/editor/jobs/integration/jobs/mera-react/branches/PR-8/config.xml
2023-10-27 12:09:21.100+0000 [id=31]	INFO	c.c.h.p.folder.AbstractFolder$3#call: Loading job EngineeringConsultants/SubscriptionLicensing/TrialService.ci/bbu-netsuite-documentation (4281.7%)
2023-10-27 12:09:25.678+0000 [id=31]	WARNING	c.c.h.p.folder.AbstractFolder#loadChildren: could not find file /var/jenkins_home/jobs/EngineeringConsultants/jobs/SubscriptionLicensing/jobs/TrialService.ci/branches/bugfix-22-38.ht877d.ymentfailure/config.xml
2023-10-27 12:09:31.657+0000 [id=31]	WARNING	c.c.h.p.folder.AbstractFolder#loadChildren: could not find file /var/jenkins_home/jobs/EngineeringConsultants/jobs/SubscriptionLicensing/jobs/TrialService.ci/branches/feat-BIIT-1474-ville.1qlj4b/config.xml
2023-10-27 12:09:33.107+0000 [id=31]	WARNING	c.c.h.p.folder.AbstractFolder#loadChildren: could not find file /var/jenkins_home/jobs/EngineeringConsultants/jobs/SubscriptionLicensing/jobs/TrialService.ci/branches/feature-sf-l.av7gml.er-tcon-trls/config.xml

I had a colleague who was using Jenkins plugins related to cleanup and housekeeping.

My suspicious is that some of the workspaces are cleaned up, however Jenkins itself is still looking for those folders.

Is there a way to some how REINDEX Jenkins with the actual directories in $JENKINS_HOME and to not search for them.

Jenkins setup:

Jenkins hosted on EKS using EFS drive mounted in $JENKINS_HOME
The server is pretty up to date:

Jenkins version: jenkins/jenkins:2.401.1-lts-jdk11
Jenkins plugins:

    - kubernetes:3900.va_dce992317b_4
    - workflow-aggregator:596.v8c21c963d92d
    - workflow-job:1308.v58d48a_763b_31
    - configuration-as-code:1647.ve39ca_b_829b_42

    - active-directory:2.25.1
    - aws-secrets-manager-credentials-provider:1.202.ve0ec0c17611c
    - credentials-binding:604.vb_64480b_c56ca_
    - role-strategy:633.v836e5b_3e80a_5
    - bitbucket:223.vd12f2bca5430
    - git:5.1.0
    - blueocean:1.27.4
    - blueocean-autofavorite:1.2.5
    # blueocean depnedency plugins
    - github-branch-source:1728.v859147241f49
    - blueocean-bitbucket-pipeline:1.27.4
    - blueocean-pipeline-editor:1.27.4
    - blueocean-pipeline-editor:1.27.4
    - blueocean-git-pipeline:1.27.4
    - blueocean-pipeline-api-impl:1.27.4
    - jjwt-api:0.11.5-77.v646c772fddb_0

    - pipeline-aws:1.43
    - pipeline-utility-steps:2.15.4
    - pipeline-model-definition:2.2141.v5402e818a_779
    - pipeline-groovy-lib:656.va_a_ceeb_6ffb_f7

    - nunit:394.v65c09a_8408b_a_
    - junit:1207.va_09d5100410f
    - basic-branch-build-strategies:71.vc1421f89888e
    - slack:631.v40deea_40323b
    - checks-api:2.0.0
    - cloudbees-bitbucket-branch-source:805.v7f97d29dc0f5
    - configuration-as-code-secret-ssm:1.0.1
    - mask-passwords:150.vf80d33113e80
    - multibranch-build-strategy-extension:1.0.10
    - performance:928.vdea_0dca_55446
    # The current version of this plugin contains a vulnerability
    - powershell:2.0
    - strict-crumb-issuer:2.1.1
    # - extended-choice-parameter:0.82
    # The current version of this plugin contains a vulnerability
    - git-changelog:3.30

    # # monitoring and cleanup
    - prometheus:2.2.3
    - cloudbees-disk-usage-simple:182.v62ca_0c992a_f3
    - ws-cleanup:0.45

    - maven-plugin:3.22
    - config-file-provider:938.ve2b_8a_591c596

    # jenkins Performance Plugins
    # Monitoring of the performance of Jenkins itself with JavaMelody.
    - monitoring:1.94.1
    # This plugin show information about jobs, slaves and the queue in Jenkins.
    - jqs-monitoring:1.4
    # Jenkins plugin to monitor pull requests with a customizable dashboard
    - pull-request-monitoring:1.7.8
    # This plugin records disk usage.
    - disk-usage:1.0
    # Remove stalled items waiting in Jenkins queue
    - queue-cleanup:1.4
    # This plugin remove old runs, depending on the orphaned item strategy configured for the pipeline, or when a branch is deleted
    - multibranch-action-triggers:1.8.6
    # The PerfPublisher plug-in scans for xml tests reports defined in the administration panel and generates trend graphs, computes stats and underline regressions and modifications.
    - perfpublisher:8.09
    - maintenance-jobs-scheduler:0.1.0
    # This plugin performs checks before and after every build to ensure that resources (e.g. open ports) are not leaked.
    - housekeeper:1.1
    # This plugin provides new post-build step named 'Discard Old Builds'(where user can select to remove unstable, Aborted builds from the build list) appears.
    - discard-old-build:1.07
    # plugin to help regulate the number of concurrent builds (This plugin allows for throttling the number of concurrent builds of a project running per node or globally)
    - throttle-concurrents:2.13
    # This plugin and its dependencies form a suite of plugins that lets you orchestrate automation
    - pipeline-utility-steps:2.15.4
    # Scriptler allows you to store/edit groovy scripts and execute it on any of the agents/nodes no need to copy/paste groovy code
    - scriptler:3.5
    # This plugin adds the ability to directly execute Groovy code.
    - groovy:453.vcdb_a_c5c99890
    # This plugin supports to run the DSL scripts
    - job-dsl:1.84
    # This plugin allows configuring projects to run with specified authorization
    - authorize-project:1.7.0
    # This plugin contributes a new view implementation that provides a dashboard / portal-like view for your Jenkins instance.
    - dashboard-view:2.487.vcf0ff9008a_c0
    # This plugin collect telemetry data(metrics, traces, logs) of remoting module with OpenTelemetry.
    - remoting-opentelemetry:1.0-rc89.d67d14d05962
    # Confluence Publisher - This plugin allows you to publish build artifacts as attachments to an Atlassian Confluence wiki page.
    - confluence-publisher:156.vf3597ca_9cf27
    # Jenkins Liquibase Runner Plugin - This Plugin Uses Liquibase version 3.8.9 - Adds liquibase changeset evaluation as an available build step
    - liquibase-runner:1.4.10
    # Build Monitor View - provides a highly visible view of the status of selected Jenkins jobs.
    - build-monitor-plugin:1.14-717.v3efcdffe8d58
    # Pipeline Graph View - This plugin aims to bring the best of Blue Ocean into the regular Jenkins UIb
    - pipeline-graph-view:191.vc6da_9d3eb_70a
    # flaky tests - This plugin helps to detect flaky tests (Currently most features of this plugin are designed for Git and Maven projects)
    - flaky-test-handler:1.2.2
    # metrics - This plugin provides Health checks on disk-space, plugins, temporary-space, thread-deadlock
    - metrics:4.2.18-439.v86a_20b_a_8318b_
    # Test Results Analyzer - This plugin shows history of test execution results in a tabular format
    - test-results-analyzer:0.4.0
    # Gitlab Plugin - installed as a safeguard for any builds that need to stay in jenkins past migration
    - gitlab-branch-source:664.v877fdc293c89
    # Parameterized Scheduler Plugin - Plugin to support setting parameters in the build schedule
    - parameterized-scheduler:255.v73827fcdf618

Thanks,
Valeri

2

Also, what is up with the over the top progress indication - 4281.7% !?

`INFO	c.c.h.p.folder.AbstractFolder$3#call: Loading job path/to/job  (4281.7%)`

I have seen similar issues my standalone instance when restarting, but never 4000%; maybe 300%

btw: those references are to the job’s config.xml file, the job configurations on the controller, not the workspace. Workspaces are typically on the agent nodes, not the controller, and in a /workspace/path/to/jobname folder tree

3

Hi Ian,
Thanks for the reply!

The problem seems to me that the xml config file is trying to load some obsolete paths and they do jot exist.
These obsolete paths are probably deleted by the house keeping job plugins but somehow the xml file is not updated I guess.

BR
v

4

I have no idea regarding the the root cause, but relieved to see someone else reporting a similar problem.

I will say however, we have also seen occur on restarts, some but not all. It goes all the way back to our Jenkins 1 instance, where it was much more frequent and exponentially delayed the load times. I never really gave it much pause as eventually things came (though once I recall it took maybe 20 minutes instead of 10s of seconds).

I believe it has to do with the folder plugin onLoad, but not sure what triggers it to go haywire, especially as it reports well over 100% progress, with no idea what the true upper limit is.

5

We are having the exact same issue, going over 44000% , taking 1-2 hours to boot. same setup - EKS + EFS.
did someone managed to resolve this? any help would be greatly appreciated

6

Unfortunately,

I wasn’t able to find a tool to somehow reindex the workspaces, hence I wasn’t able to solve it.

Br,
v

7

Appreciate it anyways!