Jenkins performance is really slow

Using Jenkins Ask a question
,
1

bs"d

Jenkins setup: 
Jenkins: 2.426.2
OS: Linux - 6.1.0-17-cloud-amd64
Java: 17.0.9 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
ace-editor:1.1
active-directory:2.34
ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
authentication-tokens:1.53.v1c90fd9191a_b_
badge:1.9.1
bootstrap4-api:4.6.0-6
bootstrap5-api:5.3.2-3
bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9
branch-api:2.1144.v1425d1c3d5a_7
build-blocker-plugin:1.7.9
build-timeout:1.32
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.0.2
cloudbees-folder:6.858.v898218f3609d
command-launcher:107.v773860566e2e
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.11.0-95.v22a_d30ee5d36
credentials:1311.vcf0a_900b_37c2
credentials-binding:642.v737c34dea_6c2
data-tables-api:1.13.8-2
display-url-api:2.200.vb_9327d658781
docker-commons:439.va_3cb_0a_6a_fb_29
docker-workflow:572.v950f58993843
durable-task:543.v262f6a_803410
echarts-api:5.4.3-2
email-ext:2.103
font-awesome-api:6.5.1-1
git:5.2.1
git-client:4.6.0
git-server:99.va_0826a_b_cdfa_d
github:1.37.3.1
github-api:1.318-461.v7a_c09c9fa_d63
github-branch-source:1767.va_7d01ea_c7256
gradle:2.9
groovy-postbuild:228.vcdb_cf7265066
gson-api:2.10.1-15.v0d99f670e0a_7
handlebars:3.0.8
instance-identity:185.v303dc7c645f9
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.16.1-373.ve709c6871598
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jjwt-api:0.11.5-77.v646c772fddb_0
jquery-detached:1.2.1
jquery3-api:3.7.1-1
jsch:0.2.16-86.v42e010d9484b_
json-path-api:2.8.0-21.v8b_7dc8b_1037b_
junit:1256.v002534a_5f33e
ldap:711.vb_d1a_491714dc
lockable-resources:1228.v1b_2379444670
mailer:463.vedf8358e006b_
mapdb-api:1.0.9-28.vf251ce40855d
matrix-auth:3.2.1
matrix-project:822.v01b_8c85d16d2
mina-sshd-api-common:2.11.0-86.v836f585d47fa_
mina-sshd-api-core:2.11.0-86.v836f585d47fa_
momentjs:1.1.1
okhttp-api:4.11.0-157.v6852a_a_fa_ec11
pam-auth:1.10
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:689.veec561a_dee13
pipeline-input-step:477.v339683a_8d55e
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2168.vf921b_4e72c73
pipeline-model-declarative-agent:1.1.1
pipeline-model-definition:2.2168.vf921b_4e72c73
pipeline-model-extensions:2.2168.vf921b_4e72c73
pipeline-rest-api:2.34
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2168.vf921b_4e72c73
pipeline-stage-view:2.34
plain-credentials:143.v1b_df8b_d3b_e48
plugin-usage-plugin:4.2
plugin-util-api:3.8.0
popper-api:1.16.1-3
popper2-api:2.11.6-4
resource-disposer:0.23
role-strategy:689.v731678c3e0eb_
scm-api:683.vb_16722fb_b_80b_
script-security:1313.v7a_6067dc7087
slack:684.v833089650554
snakeyaml-api:2.2-111.vc6598e30cc65
ssh:2.6.1
ssh-credentials:308.ve4497b_ccd8f4
ssh-slaves:2.948.vb_8050d697fec
sshd:3.322.v159e91f6a_550
structs:325.vcb_307d2a_2782
subversion:2.17.3
timestamper:1.26
token-macro:400.v35420b_922dcb_
trilead-api:2.133.vfb_8a_7b_9c5dd1
variant:60.v7290fc0eb_b_cd
windows-slaves:1.8.1
workflow-aggregator:596.v8c21c963d92d
workflow-api:1283.v99c10937efcb_
workflow-basic-steps:1042.ve7b_140c4a_e0c
workflow-cps:3837.v305192405b_c0
workflow-cps-global-lib:609.vd95673f149b_b
workflow-durable-task-step:1313.vcb_970b_d2a_fb_3
workflow-job:1385.vb_58b_86ea_fff1
workflow-multibranch:770.v1a_d0708dd1f6
workflow-scm-step:415.v434365564324
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:865.v43e78cc44e0d
ws-cleanup:0.45

I’m using Jenkins with Dockers, (Huge Thanks to @samrocketman for his help :love_you_gesture:)
This is my docker-compose.yml file

services:
  jenkins:
    image: jenkins/jenkins:lts
    #image: jenkins/jenkins:2.319.1-jdk11
    ports:
      - "8080:8080"
    volumes:
      - /mnt/jenkins_home:/var/jenkins_home
    restart: always
  ssh-agent:
    image: jenkins/ssh-agent
    restart: always
networks:
  jenkins:

The /mnt/jenkins_home is an external AWS EBS volume,

Type: gp3
IOPS: 3000
Throughput: 125
Encrypted.

The EC2 instance:

Instance type: t3.large
vCPU: 2
Memory: 8

The Jenkins docker is behind a reverse proxy, Nginx.
We use this Jenkins to run cron jobs; in the worst scenario, we have three active users and a max CPU usage recorded was 30%.

Jenkins is super super slow, any Idea how can Improve the it?

2

Can you describe more about your jobs? Do you have lots of jobs? Or folders?

You could also run this stats script and report back.

3

bs"d

Hi @samrocketman Thanks for your reply, sorry for being late.

This is the result:
Number of GitHub Organizations: 0
Number of GitHub Projects: 0
Number of Jenkins jobs: 104
Jobs with more than one build: 101
Number of users: 31
Global total number of builds: 11098664
Global total number of pull requests executed: null
Global total number of tag releases executed: null
Count of projects by type.
FreeStyleProject: 101
WorkflowJob: 3

4

11mio runs on 100 jobs, that means you have like 100.000 runs per job on average.
Did you forget to enable a build discarder?

5

@mawinter69 Thanks for your response,

You are right, we have several jobs that run per minute and connect to different nodes.
The slowness started when we moved a “regular installation of Jenkins” to containers.

6

You have 8 GiB of memory. Did you have more memory for Jenkins before moving to a container?
In an older presentation I found the statement that memory usage of jobs starts at ~2kb, so with 11Mio builds you might run into memory problems and the jvm is very busy with garbage collection

So adding a build discarder to the jobs running every minute might solve the problem.

7

You should go to /manage/systemInfoand click there on Memory Usage to see if memory is an issue

8

Screenshot 2024-03-19 at 12.06.04
Screenshot 2024-03-19 at 12.06.041536×739 142 KB

9

I would still look into applying a build discarder to your jobs.

10

What do you mean with “build discarded” ?

11

A build discarder will delete old builds after a configurable amount of time and/or number. Configure this on your jobs.

image
image2240×552 20.9 KB

You can also install Build Discarder plugin and configure it globally for any job that hasn’t configured a build discarder itself

12

We already keep 30 days logs.

13

I took a look at the script that calculates the global total number of builds. It just sums up the highest build numbers so it is not the actual number of builds that are in jenkins, but the total number of builds that have ever been executed.

On my Jenkins it reports 19Mio, but in reality it is only like 200k
In the jobs folder in jenkins_home I executed find . -name builds -type d -exec ls -l {}\; | wc -l it’s off by 2 for each job but is a good estimate.

But if you still see the slowness I don’t know what could be the reason.

What are your startup options for java, have you set limits for memory or do you have special settings for garbage collection?