Jenkins reboots without reason or error at random times during job run

Using Jenkins Ask a question
1

I use it to run autotests using python (pytest) - I customized the Freestyle project
however, after launching, at a random moment, the job freezes, Jenkins reboots - after which the running build disappears from the list
I don’t know where to look for the problem anymore
I tried a complete reinstallation of the server and Jenkins itself
Even with a minimal set of plugins
Help please

I’m new to Jenkins setup - if any additional information is needed - can provide if you tell me where to look

Jenkins setup:

Jenkins: 2.452.1
OS: Linux - 6.8.0-31-generic
Java: 11.0.23 - Ubuntu (OpenJDK 64-Bit Server VM)
Jenkins installed by instructions - How To Install Jenkins on Ubuntu 22.04 | DigitalOcean
Running Jenkins service in default port - 8080


ant: 497.v94e7d9fffa_b_9
antisamy-markup-formatter: 162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api: 4.5.14-208.v438351942757
asm-api: 9.7-33.v4d23ef79fcc8
bootstrap5-api: 5.3.3-1
bouncycastle-api: 2.30.1.77-225.v26ea_c9455fd9
branch-api: 2.1169.va_f810c56e895
build-timeout: 1.32
caffeine-api: 3.1.8-133.v17b_1ff2e0599
checks-api: 2.2.0
cloudbees-folder: 6.928.v7c780211d66e
commons-lang3-api: 3.13.0-62.v7d18e55f51e2
commons-text-api: 1.11.0-109.vfe16c66636eb_
credentials: 1337.v60b_d7b_c7b_c9f
credentials-binding: 677.vdc9d38cb_254d
dark-theme: 439.vdef09f81f85e
display-url-api: 2.204.vf6fddd8a_8b_e9
durable-task: 555.v6802fe0f0b_82
echarts-api: 5.5.0-1
email-ext: 1814.v404722f34263
font-awesome-api: 6.5.2-1
git: 5.2.2
git-client: 4.7.0
github: 1.39.0
github-api: 1.318-461.v7a_c09c9fa_d63
github-branch-source: 1789.v5b_0c0cea_18c3
gradle: 2.12
gson-api:2.10.1-15.v0d99f670e0a_7
instance-identity:185.v303dc7c645f9
ionicons-api:74.v93d5eb_813d5f
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.9-1
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery3-api:3.7.1-2
json-api:20240303-41.v94e11e6de726
json-path-api:2.9.0-58.v62e3e85b_a_655
junit:1265.v65b_14fa_f12f0
ldap:725.v3cb_b_711b_1a_ef
mailer:472.vf7c289a_4b_420
matrix-auth:3.2.2
matrix-project:822.824.v14451b_c0fd42
metrics:4.2.21-449.v6960d7c54c69
mina-sshd-api-common:2.12.1-101.v85b_e08b_780dd
mina-sshd-api-core:2.12.1-101.v85b_e08b_780dd
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
pam-auth:1.10
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-graph-view:287.v3ef017b_780d5
pipeline-groovy-lib:710.v4b_94b_077a_808
pipeline-input-step:495.ve9c153f6067b_
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2198.v41dd8ef6dd56
pipeline-model-definition:2.2198.v41dd8ef6dd56
pipeline-model-extensions:2.2198.v41dd8ef6dd56
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2198.v41dd8ef6dd56
plain-credentials:182.v468b_97b_9dcb_8
plugin-util-api:4.1.0
resource-disposer:0.23
scm-api:690.vfc8b_54395023
script-security:1336.vf33a_a_9863911
snakeyaml-api:2.2-111.vc6598e30cc65
ssh-credentials:337.v395d2403ccd4
ssh-slaves:2.948.vb_8050d697fec
structs:337.v1b_04ea_4df7c8
theme-manager:215.vc1ff18d67920
timestamper:1.27
token-macro:400.v35420b_922dcb_
trilead-api:2.142.v748523a_76693
variant:60.v7290fc0eb_b_cd
workflow-aggregator:596.v8c21c963d92d
workflow-api:1311.v4250456a_e552
workflow-basic-steps:1058.vcb_fc1e3a_21a_9
workflow-cps:3894.3896.vca_2c931e7935
workflow-durable-task-step:1353.v1891a_b_01da_18
workflow-job:1400.v7fd111b_ec82f
workflow-multibranch:783.va_6eb_ef636fb_d
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:657.v03b_e8115821b_
workflow-support:907.v6713a_ed8a_573
ws-cleanup:0.45

image
image1920×434 145 KB

2

There are some questions open for you to answer:
Are you running the builds on dedicated agents or do you run the them directly on the controller. The latter is not recommended.
Does your machine have enough memory? From the name it looks as if you only have 2GiB of memory. So if you’re short on memory the OOM killer might jump in and kill your Jenkins process

1 Like
3

Thank you for your response

  1. Yes - it looks like I’m running them on the controller - how can I do it differently?
  2. So far I have only 2 GB - is it not enough?
4

You need to attach an agent to Jenkins that will do the work. See Using Jenkins agents

The 2 GB might be enough when you offload the work to an agent. It also depends how much jobs you have and how many runs all the jobs have in total.

You should see in the OS logs if your process was killed due to OOM.

1 Like
5

Found the logs, really OOM-kill
now I will try the option with Jenkins agents and come back with the results
Thanks a lot for the explanation

image
image2874×124 148 KB

6

did you run also your browser on that machine? Seems that a process with name chrome was killed.
Then I’m not surprised that you run OOM.

7

Can you please recommend what are the minimum server requirements for the controller and for the agent?
Will 2GiB be enough for the controller?
and how much memory will be needed for the agent if we run tests using chrome?

8

I would assume that the 2 GiB for the controller are sufficient for the beginning.
For the agent it’s hard to tell. The agent process doesn’t need much, I assume the memory requirement is dominated by your tests. So you should check if 2GiB are sufficient and if not you might need to give the VM more memory.

1 Like
9

thanks, increasing the RAM really solved the problem