# Jenkins performance is really slow

**URL:** <https://community.jenkins.io/t/jenkins-performance-is-really-slow/12015>\
**Category:** Ask a question\
**Tags:** question, docker\
**Created:** [January 25, 2024, 7:33am UTC](https://community.jenkins.io/t/jenkins-performance-is-really-slow/12015 "2024-01-25T07:33:41Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Abyweinberg](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/abyweinberg/32/6642_2.png) [@Abyweinberg](https://community.jenkins.io/u/Abyweinberg)\
**Post date:** [January 25, 2024, 7:33am UTC](https://community.jenkins.io/t/jenkins-performance-is-really-slow/12015/1 "2024-01-25T07:33:41Z")

</div>

bs"d

> **Jenkins setup:**
>
> ```
> Jenkins: 2.426.2
> OS: Linux - 6.1.0-17-cloud-amd64
> Java: 17.0.9 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
> ---
> ace-editor:1.1
> active-directory:2.34
> ant:497.v94e7d9fffa_b_9
> antisamy-markup-formatter:162.v0e6ec0fcfcf6
> apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
> authentication-tokens:1.53.v1c90fd9191a_b_
> badge:1.9.1
> bootstrap4-api:4.6.0-6
> bootstrap5-api:5.3.2-3
> bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9
> branch-api:2.1144.v1425d1c3d5a_7
> build-blocker-plugin:1.7.9
> build-timeout:1.32
> caffeine-api:3.1.8-133.v17b_1ff2e0599
> checks-api:2.0.2
> cloudbees-folder:6.858.v898218f3609d
> command-launcher:107.v773860566e2e
> commons-lang3-api:3.13.0-62.v7d18e55f51e2
> commons-text-api:1.11.0-95.v22a_d30ee5d36
> credentials:1311.vcf0a_900b_37c2
> credentials-binding:642.v737c34dea_6c2
> data-tables-api:1.13.8-2
> display-url-api:2.200.vb_9327d658781
> docker-commons:439.va_3cb_0a_6a_fb_29
> docker-workflow:572.v950f58993843
> durable-task:543.v262f6a_803410
> echarts-api:5.4.3-2
> email-ext:2.103
> font-awesome-api:6.5.1-1
> git:5.2.1
> git-client:4.6.0
> git-server:99.va_0826a_b_cdfa_d
> github:1.37.3.1
> github-api:1.318-461.v7a_c09c9fa_d63
> github-branch-source:1767.va_7d01ea_c7256
> gradle:2.9
> groovy-postbuild:228.vcdb_cf7265066
> gson-api:2.10.1-15.v0d99f670e0a_7
> handlebars:3.0.8
> instance-identity:185.v303dc7c645f9
> ionicons-api:56.v1b_1c8c49374e
> jackson2-api:2.16.1-373.ve709c6871598
> jakarta-activation-api:2.0.1-3
> jakarta-mail-api:2.0.1-3
> javax-activation-api:1.2.0-6
> javax-mail-api:1.6.2-9
> jaxb:2.3.9-1
> jdk-tool:73.vddf737284550
> jjwt-api:0.11.5-77.v646c772fddb_0
> jquery-detached:1.2.1
> jquery3-api:3.7.1-1
> jsch:0.2.16-86.v42e010d9484b_
> json-path-api:2.8.0-21.v8b_7dc8b_1037b_
> junit:1256.v002534a_5f33e
> ldap:711.vb_d1a_491714dc
> lockable-resources:1228.v1b_2379444670
> mailer:463.vedf8358e006b_
> mapdb-api:1.0.9-28.vf251ce40855d
> matrix-auth:3.2.1
> matrix-project:822.v01b_8c85d16d2
> mina-sshd-api-common:2.11.0-86.v836f585d47fa_
> mina-sshd-api-core:2.11.0-86.v836f585d47fa_
> momentjs:1.1.1
> okhttp-api:4.11.0-157.v6852a_a_fa_ec11
> pam-auth:1.10
> pipeline-build-step:540.vb_e8849e1a_b_d8
> pipeline-github-lib:42.v0739460cda_c4
> pipeline-graph-analysis:202.va_d268e64deb_3
> pipeline-groovy-lib:689.veec561a_dee13
> pipeline-input-step:477.v339683a_8d55e
> pipeline-milestone-step:111.v449306f708b_7
> pipeline-model-api:2.2168.vf921b_4e72c73
> pipeline-model-declarative-agent:1.1.1
> pipeline-model-definition:2.2168.vf921b_4e72c73
> pipeline-model-extensions:2.2168.vf921b_4e72c73
> pipeline-rest-api:2.34
> pipeline-stage-step:305.ve96d0205c1c6
> pipeline-stage-tags-metadata:2.2168.vf921b_4e72c73
> pipeline-stage-view:2.34
> plain-credentials:143.v1b_df8b_d3b_e48
> plugin-usage-plugin:4.2
> plugin-util-api:3.8.0
> popper-api:1.16.1-3
> popper2-api:2.11.6-4
> resource-disposer:0.23
> role-strategy:689.v731678c3e0eb_
> scm-api:683.vb_16722fb_b_80b_
> script-security:1313.v7a_6067dc7087
> slack:684.v833089650554
> snakeyaml-api:2.2-111.vc6598e30cc65
> ssh:2.6.1
> ssh-credentials:308.ve4497b_ccd8f4
> ssh-slaves:2.948.vb_8050d697fec
> sshd:3.322.v159e91f6a_550
> structs:325.vcb_307d2a_2782
> subversion:2.17.3
> timestamper:1.26
> token-macro:400.v35420b_922dcb_
> trilead-api:2.133.vfb_8a_7b_9c5dd1
> variant:60.v7290fc0eb_b_cd
> windows-slaves:1.8.1
> workflow-aggregator:596.v8c21c963d92d
> workflow-api:1283.v99c10937efcb_
> workflow-basic-steps:1042.ve7b_140c4a_e0c
> workflow-cps:3837.v305192405b_c0
> workflow-cps-global-lib:609.vd95673f149b_b
> workflow-durable-task-step:1313.vcb_970b_d2a_fb_3
> workflow-job:1385.vb_58b_86ea_fff1
> workflow-multibranch:770.v1a_d0708dd1f6
> workflow-scm-step:415.v434365564324
> workflow-step-api:639.v6eca_cd8c04a_a_
> workflow-support:865.v43e78cc44e0d
> ws-cleanup:0.45
> ```

I’m using Jenkins with Dockers, (Huge Thanks to @samrocketman for his help 🤟)  
This is my `docker-compose.yml` file

```auto
services:
  jenkins:
    image: jenkins/jenkins:lts
    #image: jenkins/jenkins:2.319.1-jdk11
    ports:
      - "8080:8080"
    volumes:
      - /mnt/jenkins_home:/var/jenkins_home
    restart: always
  ssh-agent:
    image: jenkins/ssh-agent
    restart: always
networks:
  jenkins:

```

The `/mnt/jenkins_home` is an external AWS EBS volume,

```auto
Type: gp3
IOPS: 3000
Throughput: 125
Encrypted.

```

The EC2 instance:

```auto
Instance type: t3.large
vCPU: 2
Memory: 8

```

The Jenkins docker is behind a reverse proxy, Nginx.  
We use this Jenkins to run cron jobs; in the worst scenario, we have three active users and a max CPU usage recorded was 30%.

Jenkins is super super slow, any Idea how can Improve the it?

---

<div class="post-metadata">

**Author:** ![samrocketman](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/samrocketman/32/4631_2.png) [@samrocketman](https://community.jenkins.io/u/samrocketman)\
**Post date:** [January 29, 2024, 2:54am UTC](https://community.jenkins.io/t/jenkins-performance-is-really-slow/12015/2 "2024-01-29T02:54:14Z")

</div>

Can you describe more about your jobs? Do you have lots of jobs? Or folders?

You could also run [this stats script](https://github.com/samrocketman/jenkins-script-console-scripts/blob/02539f0c088daf7aab8b40f00c59313abcabe17f/jenkins-stats.groovy) and report back.

---

<div class="post-metadata">

**Author:** ![Abyweinberg](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/abyweinberg/32/6642_2.png) [@Abyweinberg](https://community.jenkins.io/u/Abyweinberg)\
**Post date:** [March 19, 2024, 7:51am UTC](https://community.jenkins.io/t/jenkins-performance-is-really-slow/12015/3 "2024-03-19T07:51:30Z")

</div>

bs"d

Hi @samrocketman Thanks for your reply, sorry for being late.

This is the result:  
Number of GitHub Organizations: 0  
Number of GitHub Projects: 0  
Number of Jenkins jobs: 104  
Jobs with more than one build: 101  
Number of users: 31  
Global total number of builds: 11098664  
Global total number of pull requests executed: null  
Global total number of tag releases executed: null  
Count of projects by type.  
FreeStyleProject: 101  
WorkflowJob: 3

---

<div class="post-metadata">

**Author:** ![mawinter69](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/mawinter69/32/1625_2.png) [@mawinter69](https://community.jenkins.io/u/mawinter69)\
**Post date:** [March 19, 2024, 8:50am UTC](https://community.jenkins.io/t/jenkins-performance-is-really-slow/12015/4 "2024-03-19T08:50:21Z")

</div>

11mio runs on 100 jobs, that means you have like 100.000 runs per job on average.  
Did you forget to enable a build discarder?

---

<div class="post-metadata">

**Author:** ![Abyweinberg](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/abyweinberg/32/6642_2.png) [@Abyweinberg](https://community.jenkins.io/u/Abyweinberg)\
**Post date:** [March 19, 2024, 9:27am UTC](https://community.jenkins.io/t/jenkins-performance-is-really-slow/12015/5 "2024-03-19T09:27:33Z")

</div>

@mawinter69 Thanks for your response,

You are right, we have several jobs that run per minute and connect to different nodes.  
The slowness started when we moved a “regular installation of Jenkins” to containers.

---

<div class="post-metadata">

**Author:** ![mawinter69](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/mawinter69/32/1625_2.png) [@mawinter69](https://community.jenkins.io/u/mawinter69)\
**Post date:** [March 19, 2024, 9:49am UTC](https://community.jenkins.io/t/jenkins-performance-is-really-slow/12015/6 "2024-03-19T09:49:28Z")

</div>

You have 8 GiB of memory. Did you have more memory for Jenkins before moving to a container?  
In an older [presentation](https://assets.ctfassets.net/vtn4rfaw6n2j/2016-jenkins-world-soyouwanttobuildtheworldslargestjenkinscluste/9861b86403e5dc3c81bbc13e129bf9e9/2016-jenkins-world-soyouwanttobuildtheworldslargestjenkinscluster_final.pdf) I found the statement that memory usage of jobs starts at ~2kb, so with 11Mio builds you might run into memory problems and the jvm is very busy with garbage collection

So adding a build discarder to the jobs running every minute might solve the problem.

---

<div class="post-metadata">

**Author:** ![mawinter69](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/mawinter69/32/1625_2.png) [@mawinter69](https://community.jenkins.io/u/mawinter69)\
**Post date:** [March 19, 2024, 9:57am UTC](https://community.jenkins.io/t/jenkins-performance-is-really-slow/12015/7 "2024-03-19T09:57:13Z")

</div>

You should go to `/manage/systemInfo`and click there on `Memory Usage` to see if memory is an issue

---

<div class="post-metadata">

**Author:** ![Abyweinberg](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/abyweinberg/32/6642_2.png) [@Abyweinberg](https://community.jenkins.io/u/Abyweinberg)\
**Post date:** [March 19, 2024, 10:14am UTC](https://community.jenkins.io/t/jenkins-performance-is-really-slow/12015/8 "2024-03-19T10:14:47Z")

</div>

> [@mawinter69](#):
>
> You should go to `/manage/systemInfo`and click there on `Memory Usage` to see if memory is an issue
> 
> This is the usage.

 ![Screenshot 2024-03-19 at 12.06.04](https://europe1.discourse-cdn.com/flex013/uploads/jenkins/original/2X/8/8f7dfb877082eede4476226405522944bdfd9d79.png)

---

<div class="post-metadata">

**Author:** ![mawinter69](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/mawinter69/32/1625_2.png) [@mawinter69](https://community.jenkins.io/u/mawinter69)\
**Post date:** [March 19, 2024, 3:50pm UTC](https://community.jenkins.io/t/jenkins-performance-is-really-slow/12015/9 "2024-03-19T15:50:39Z")

</div>

I would still look into applying a build discarder to your jobs.

---

<div class="post-metadata">

**Author:** ![Abyweinberg](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/abyweinberg/32/6642_2.png) [@Abyweinberg](https://community.jenkins.io/u/Abyweinberg)\
**Post date:** [March 20, 2024, 9:10am UTC](https://community.jenkins.io/t/jenkins-performance-is-really-slow/12015/10 "2024-03-20T09:10:32Z")

</div>

What do you mean with “build discarded” ?

---

<div class="post-metadata">

**Author:** ![mawinter69](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/mawinter69/32/1625_2.png) [@mawinter69](https://community.jenkins.io/u/mawinter69)\
**Post date:** [March 20, 2024, 9:30am UTC](https://community.jenkins.io/t/jenkins-performance-is-really-slow/12015/11 "2024-03-20T09:30:01Z")

</div>

A build discarder will delete old builds after a configurable amount of time and/or number. Configure this on your jobs.

 ![image](https://europe1.discourse-cdn.com/flex013/uploads/jenkins/original/2X/8/85e4d4366539b20e1a2fa7ff7c955e9292487f36.png)

You can also install [Build Discarder](https://plugins.jenkins.io/build-discarder/) plugin and configure it globally for any job that hasn’t configured a build discarder itself

---

<div class="post-metadata">

**Author:** ![Abyweinberg](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/abyweinberg/32/6642_2.png) [@Abyweinberg](https://community.jenkins.io/u/Abyweinberg)\
**Post date:** [March 20, 2024, 2:20pm UTC](https://community.jenkins.io/t/jenkins-performance-is-really-slow/12015/12 "2024-03-20T14:20:32Z")

</div>

We already keep 30 days logs.

---

<div class="post-metadata">

**Author:** ![mawinter69](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/mawinter69/32/1625_2.png) [@mawinter69](https://community.jenkins.io/u/mawinter69)\
**Post date:** [March 20, 2024, 5:51pm UTC](https://community.jenkins.io/t/jenkins-performance-is-really-slow/12015/13 "2024-03-20T17:51:00Z")

</div>

I took a look at the script that calculates the global total number of builds. It just sums up the highest build numbers so it is not the actual number of builds that are in jenkins, but the total number of builds that have ever been executed.

On my Jenkins it reports 19Mio, but in reality it is only like 200k  
In the jobs folder in jenkins\_home I executed `find . -name builds -type d -exec ls -l {}\; | wc -l` it’s off by 2 for each job but is a good estimate.

But if you still see the slowness I don’t know what could be the reason.

What are your startup options for java, have you set limits for memory or do you have special settings for garbage collection?