# Speeding up SCM-Pipelines **URL:** \ **Category:** Ask a question\ **Created:** [June 26, 2024, 1:26pm UTC](https://community.jenkins.io/t/speeding-up-scm-pipelines/16383 "2024-06-26T13:26:27Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![GeneralMidiRotkiv](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/generalmidirotkiv/32/8791_2.png) [@GeneralMidiRotkiv](https://community.jenkins.io/u/GeneralMidiRotkiv)\ **Post date:** [June 26, 2024, 1:26pm UTC](https://community.jenkins.io/t/speeding-up-scm-pipelines/16383/1 "2024-06-26T13:26:27Z")
Hey there! I am pretty new to Jenkins but have been getting deeper and deeper into it ever since I took an existing setup over from a colleague who left our company. What my Jenkins does is find all files of a certain type inside a specified folder and use a 3rd party software on each of the found files. Since the amount of found files and therefore also the created jobs can go into the thousands, I have been working on minimising the execution time of each of those jobs but I’m currently facing an issue where I can’t seem to get to a satisfying solution. The thing is that I am using gitHub to store all my pipeline scripts and I am chaining three pipelines together: **Pipeline 1** finds all the files and stores their paths in a bunch of .txt files, then activates Pipeline 2. **Pipeline 2** takes these files, goes through them one by one and starts Pipeline 3 on each filepath in the current file. It then waits for the current Pipeline 3 job to finish, then goes onto the next filepath and so on. **Pipeline 3** takes the filepath and starts the 3rd party tool. This is what is being done many thousand times. The problem is that currently I have all three pipelines set up to clone from gitHub and in case of Pipelines 1 and 2 that’s not a big deal because these run only once. However, since Pipeline 3 runs so many times and since it currently contacts gitHub and then clones the repo, there are at least 7 seconds in between the start of the pipeline and the actual work that I want performed wasted. **tl;dr:** **What I would like to have: I’d like to find a way how to only clone the remote once for every start of Pipeline 1, so that Pipeline 3 does not need to get a current copy of the remote for every started job. This will save so much time and since I won’t change the pipeline code that often, fetching the repo anew for every run of Pipeline 3 is absolutely unnecessary.** Any help on this one would be greatly appreciated! ## Jenkins setup: Jenkins: 2.452.2 OS: Windows 10 - 10.0 Java: 17.0.9 - Azul Systems, Inc. (OpenJDK 64-Bit Server VM) ant:497.v94e7d9fffa\_b\_9 antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 asm-api:9.7-33.v4d23ef79fcc8 bootstrap5-api:5.3.3-1 bouncycastle-api:2.30.1.78.1-233.vfdcdeb\_0a\_08a\_a\_ branch-api:2.1169.va\_f810c56e895 build-timeout:1.32 caffeine-api:3.1.8-133.v17b\_1ff2e0599 checks-api:2.2.0 cloudbees-folder:6.928.v7c780211d66e command-launcher:107.v773860566e2e commons-lang3-api:3.14.0-76.vda\_5591261cfe commons-text-api:1.12.0-119.v73ef73f2345d credentials:1337.v60b\_d7b\_c7b\_c9f credentials-binding:677.vdc9d38cb\_254d dark-theme:439.vdef09f81f85e display-url-api:2.204.vf6fddd8a\_8b\_e9 durable-task:555.v6802fe0f0b\_82 echarts-api:5.5.0-1 eddsa-api:0.3.0-4.v84c6f0f4969e email-ext:1814.v404722f34263 font-awesome-api:6.5.2-1 git:5.2.2 git-client:5.0.0 github:1.39.0 github-api:1.318-461.v7a\_c09c9fa\_d63 github-branch-source:1789.v5b\_0c0cea\_18c3 gradle:2.12 gson-api:2.11.0-41.v019fcf6125dc http\_request:1.18 instance-identity:185.v303dc7c645f9 ionicons-api:74.v93d5eb\_813d5f jackson2-api:2.17.0-379.v02de8ec9f64c jakarta-activation-api:2.1.3-1 jakarta-mail-api:2.1.3-1 javax-activation-api:1.2.0-7 javax-mail-api:1.6.2-10 jaxb:2.3.9-1 jdk-tool:73.vddf737284550 jjwt-api:0.11.5-112.ve82dfb\_224b\_a\_d joda-time-api:2.12.7-29.v5a\_b\_e3a\_82269a\_ jquery:1.12.4-1 jquery3-api:3.7.1-2 json-api:20240303-41.v94e11e6de726 json-path-api:2.9.0-58.v62e3e85b\_a\_655 junit:1265.v65b\_14fa\_f12f0 ldap:725.v3cb\_b\_711b\_1a\_ef locale:511.v212370760160 mailer:472.vf7c289a\_4b\_420 matrix-auth:3.2.2 matrix-project:832.va\_66e270d2946 metrics:4.2.21-451.vd51df8df52ec mina-sshd-api-common:2.12.1-113.v4d3ea\_5eb\_7f72 mina-sshd-api-core:2.12.1-113.v4d3ea\_5eb\_7f72 okhttp-api:4.11.0-172.vda\_da\_1feeb\_c6e pam-auth:1.11 pipeline-build-step:540.vb\_e8849e1a\_b\_d8 pipeline-github-lib:61.v629f2cc41d83 pipeline-graph-analysis:216.vfd8b\_ece330ca\_ pipeline-graph-view:287.v3ef017b\_780d5 pipeline-groovy-lib:710.v4b\_94b\_077a\_808 pipeline-input-step:495.ve9c153f6067b\_ pipeline-milestone-step:119.vdfdc43fc3b\_9a\_ pipeline-model-api:2.2198.v41dd8ef6dd56 pipeline-model-definition:2.2198.v41dd8ef6dd56 pipeline-model-extensions:2.2198.v41dd8ef6dd56 pipeline-stage-step:312.v8cd10304c27a\_ pipeline-stage-tags-metadata:2.2198.v41dd8ef6dd56 plain-credentials:182.v468b\_97b\_9dcb\_8 plugin-util-api:4.1.0 resource-disposer:0.23 scm-api:690.vfc8b\_54395023 script-security:1341.va\_2819b\_414686 snakeyaml-api:2.2-111.vc6598e30cc65 ssh-credentials:337.v395d2403ccd4 ssh-slaves:2.968.v6f8823c91de4 sshd:3.330.vc866a\_8389b\_58 structs:337.v1b\_04ea\_4df7c8 theme-manager:262.vc57ee4a\_eda\_5d timestamper:1.27 token-macro:400.v35420b\_922dcb\_ trilead-api:2.147.vb\_73cc728a\_32e variant:60.v7290fc0eb\_b\_cd workflow-aggregator:596.v8c21c963d92d workflow-api:1316.v33eb\_726c50b\_a\_ workflow-basic-steps:1058.vcb\_fc1e3a\_21a\_9 workflow-cps:3894.3896.vca\_2c931e7935 workflow-durable-task-step:1353.v1891a\_b\_01da\_18 workflow-job:1400.v7fd111b\_ec82f workflow-multibranch:783.va\_6eb\_ef636fb\_d workflow-scm-step:427.v4ca\_6512e7df1 workflow-step-api:657.v03b\_e8115821b\_ workflow-support:907.v6713a\_ed8a\_573 ws-cleanup:0.46 ---
**Author:** ![mawinter69](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/mawinter69/32/1625_2.png) [@mawinter69](https://community.jenkins.io/u/mawinter69)\ **Post date:** [June 26, 2024, 1:31pm UTC](https://community.jenkins.io/t/speeding-up-scm-pipelines/16383/2 "2024-06-26T13:31:38Z")
merge the code from pipeline 3 into pipeline 2 ---
**Author:** ![kikmon](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/kikmon/32/2837_2.png) [@kikmon](https://community.jenkins.io/u/kikmon)\ **Post date:** [June 27, 2024, 3:33am UTC](https://community.jenkins.io/t/speeding-up-scm-pipelines/16383/3 "2024-06-27T03:33:08Z")
To my knowledge there is no way of avoiding the git tax when using an scm pipeline. It would be great if Jenkins controller could maintain some kind of local mirror of the github pipeline source, and refresh it in the background. Agents could pull their config from the controller instead of poking directly the original repo, and it would help a lot reduce the git tax, and be more reliable to communication errors with the real source. Do anyone know if such feature is planned in future Jenkins versions ? ---
**Author:** ![sodul](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/sodul/32/7828_2.png) [@sodul](https://community.jenkins.io/u/sodul)\ **Post date:** [June 27, 2024, 4:23am UTC](https://community.jenkins.io/t/speeding-up-scm-pipelines/16383/4 "2024-06-27T04:23:49Z")
> [@GeneralMidiRotkiv](#): > > However, since Pipeline 3 runs so many times and since it currently contacts gitHub and then clones the repo, there are at least 7 seconds in between the start of the pipeline and the actual work that I want performed wasted. > > **tl;dr:** > **What I would like to have: I’d like to find a way how to only clone the remote once for every start of Pipeline 1, so that Pipeline 3 does not need to get a current copy of the remote for every started job. This will save so much time and since I won’t change the pipeline code that often, fetching the repo anew for every run of Pipeline 3 is absolutely unnecessary.** > > Any If your agents are persistent and not ephemeral, use a reference clone. The way it works is that you have yet an other pipeline that makes a clone of the git repo in an absolute path that your 3rd pipeline will be able to access. Here is a blog post about this. > **[Use Git Reference Repository to Reduce Jenkins’ Disk Space](https://argus-sec.com/blog/engineering-blog/how-you-can-use-git-reference-repository-to-reduce-jenkins-disk-space/)** > > Read how you can use Git Reference Repository to reduce Jenkins’ disk space and what are the cyber security benefits. Click here & learn more We run jenkins on Kubernetes so all the agents are ephemeral but we plan on attaching an EFS volume to address this. Our controllers have persistent volumes and they do use reference clones and sparse checkouts since they only need to see the most recent jenkinsfiles which are all under one folder in our case. If you cannot do a reference clone, try to do a shallow clone so you do not pull the entire git history and finally if possible do a sparse checkout. ---
**Author:** ![GeneralMidiRotkiv](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/generalmidirotkiv/32/8791_2.png) [@GeneralMidiRotkiv](https://community.jenkins.io/u/GeneralMidiRotkiv)\ **Post date:** [July 1, 2024, 7:12am UTC](https://community.jenkins.io/t/speeding-up-scm-pipelines/16383/5 "2024-07-01T07:12:09Z")
Thanks for your replies, but none of them were the solution to my issue. However, the [website](https://argus-sec.com/blog/engineering-blog/how-you-can-use-git-reference-repository-to-reduce-jenkins-disk-space/) linked by @sodul put me on the right path. Here is what I did: - 1. Add some code to pipeline 1 so that it clones the remote repo once into a local folder in the workspace folder. I am using declarative pipelines, so here is the code for that: 2. Set up Pipeline 2 and 3 so that they get access to the local repo freshly cloned by Pipeline 1: 3. prevent pipelines 2 and 3 from doing the default checkout done at the start of a scm-pipeline by inserting the following line of code into the pipeline code: And that’s it! Now pipelines 2 and 3 use the locally stored clone of the remote that has been freshly updated by pipeline 1 beforehand, instead of re-cloning every time they run. ---
**Author:** ![sodul](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/sodul/32/7828_2.png) [@sodul](https://community.jenkins.io/u/sodul)\ **Post date:** [July 2, 2024, 11:07pm UTC](https://community.jenkins.io/t/speeding-up-scm-pipelines/16383/6 "2024-07-02T23:07:21Z")
Good to hear you found a way to make it work for you. Each situation can be a bit unique and can require a custom solution. The good news is that Jenkins is very flexible which makes most problems solvable.