Why are builds appearing on built-in node but it's not using the executors assigned (causing high memory issue)

Using Jenkins Ask a question
1

We have been having lots of memory issue with Jenkins (restarting it worked fine for a few days). We even allocated 32G of memory but it still eventually run out.

We have noticed there are many builds that are appearing on the built-in node, but we did restrict those builds to run on agents, which is probably why in the screenshot, it’s not using the 2 executors of the built-in node, but those builds are still appearing on built-in node, as well as on the agent.

We are wondering if this is causing the memory issue and any help would be greatly appreciated.

built-in node
built-in node1951×925 263 KB

Jenkins setup:
Jenkins: 2.387.1
OS: Linux - 5.4.0-162-generic
Java: 11.0.20.1 - Ubuntu (OpenJDK 64-Bit Server VM)

Office-365-Connector:4.18.0
Parameterized-Remote-Trigger:3.1.6.3
PrioritySorter:4.1.0
WebSVN2:0.9
ace-editor:1.1
active-directory:2.29
analysis-core:1.96
ansible:1.1
ansicolor:1.0.2
ant:481.v7b_09e538fcca
antisamy-markup-formatter:155.v795fb_8702324
apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61
artifactory:3.18.0
audit-trail:3.11
authentication-tokens:1.4
authorize-project:1.4.0
badge:1.9.1
bootstrap4-api:4.6.0-5
bootstrap5-api:5.2.1-3
bouncycastle-api:2.26
branch-api:2.1051.v9985666b_f6cc
build-blocker-plugin:1.7.8
build-failure-analyzer:1.27.1
build-monitor-plugin:1.13+build.202205140447
build-timeout:1.28
build-user-vars-plugin:1.8
caffeine-api:2.9.3-65.v6a_47d0f4d1fe
checks-api:1.8.1
claim:2.18.2
cloudbees-folder:6.6.800.v71307ca_b_986b-4-gb149626-SNAPSHOT (private-0ec0a7ec-hteng)
command-launcher:90.v669d7ccb_7c31
commons-httpclient3-api:3.1-3
commons-lang3-api:3.12.0-36.vd97de6465d5b_
commons-text-api:1.10.0-27.vb_fa_3896786a_7
conditional-buildstep:1.4.2
config-file-provider:3.11.1
configurationslicing:430.v966357576543
copyartifact:1.48
credentials:1214.v1de940103927
credentials-binding:523.vd859a_4b_122e6
customize-build-now:1.1
cvs:2.19.1
dashboard-view:2.466.vdfefd95a_b_f8d
data-tables-api:1.12.1-4
database:148.v4a_ff2ca_608b_7
database-mysql:1.4
date-parameter:0.0.4
display-url-api:2.3.7
docker-build-step:2.8
docker-commons:1.21
docker-java-api:3.1.5.2
docker-plugin:1.2.9
docker-workflow:563.vd5d2e5c4007f
durable-task:503.v57154d18d478
echarts-api:5.4.0-1
email-ext:2.93
emailext-template:1.5
envinject:2.892.v25453b_80e595
envinject-api:1.199.v3ce31253ed13
extended-choice-parameter:359.v35dcfdd0c20d
external-monitor-job:203.v683c09d993b_9
extra-columns:1.25
font-awesome-api:6.2.1-1
git:5.0.0
git-client:4.0.0
git-server:99.va_0826a_b_cdfa_d
gitlab-api:1.0.6
gitlab-plugin:1.6.0
gradle:2.3
greenballs:1.15.1
groovy:453.vcdb_a_c5c99890
groovy-postbuild:2.5
handlebars:3.0.8
hidden-parameter:0.0.5
htmlpublisher:1.31
http_request:1.12
icon-shim:3.0.0
instance-identity:142.v04572ca_5b_265
ionicons-api:31.v4757b_6987003
jackson2-api:2.14.1-313.v504cdd45c18b
jakarta-activation-api:2.0.1-2
jakarta-mail-api:2.0.1-2
javadoc:226.v71211feb_e7e9
javax-activation-api:1.2.0-5
javax-mail-api:1.6.2-8
jaxb:2.3.7-1
jdk-tool:63.v62d2fd4b_4793
jersey2-api:2.38-1
jfrog:1.0.5
jobConfigHistory:1198.v4d5736c2308c
jquery:1.12.4-1
jquery-detached:1.2.1
jquery-ui:1.0.2
jquery3-api:3.6.1-2
jsch:0.1.55.61.va_e9ee26616e7
junit:1166.va_436e268e972
ldap:659.v8ca_b_a_fe79fa_d
leastload:3.0.0
lockable-resources:1109.v470b_a_941b_06e
mailer:438.v02c7f0a_12fa_4
managed-scripts:1.5.6
mapdb-api:1.0.9-28.vf251ce40855d
mask-passwords:3.3
matrix-auth:3.1.5
matrix-combinations-parameter:1.3.2
matrix-project:785-SNAPSHOT (private-7a57a3bc-hteng)
maven-plugin:3.20
mina-sshd-api-common:2.9.2-50.va_0e1f42659a_a
mina-sshd-api-core:2.9.2-50.va_0e1f42659a_a
momentjs:1.1.1
monitoring:1.91.0
multi-branch-project-plugin:0.7-SNAPSHOT (private-8749496e-devops)
mysql-api:8.0.16
next-build-number:1.8
nodelabelparameter:1.11.0
pam-auth:1.10
parameterized-trigger:2.45
pipeline-build-step:2.18
pipeline-graph-analysis:195.v5812d95a_a_2f9
pipeline-groovy-lib:629.vb_5627b_ee2104
pipeline-input-step:466.v6d0a_5df34f81
pipeline-milestone-step:101.vd572fef9d926
pipeline-model-api:2.2118.v31fd5b_9944b_5
pipeline-model-declarative-agent:1.1.1
pipeline-model-definition:2.2118.v31fd5b_9944b_5
pipeline-model-extensions:2.2118.v31fd5b_9944b_5
pipeline-rest-api:2.28
pipeline-stage-step:296.v5f6908f017a_5
pipeline-stage-tags-metadata:2.2118.v31fd5b_9944b_5
pipeline-stage-view:2.28
pipeline-utility-steps:2.14.0
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:2.20.0
popper-api:1.16.1-3
popper2-api:2.11.6-2
postbuildscript:2.11.0
preSCMbuildstep:44.v6ef4fd97f56e
project-stats-plugin:23.v47fee1f77b_84
promoted-builds:867-0117patch-SNAPSHOT (private-da9193b6-hteng)
publish-over:0.22
publish-over-cifs:0.16
publish-over-ftp:1.17
publish-over-ssh:1.24
rebuild:1.34
resource-disposer:0.20
robot:3.3.0
role-strategy:584.vf8e515397ecd-SNAPSHOT (private-1dc36647-hteng)
run-condition:1.5
saferestart:0.7
saml:2.298.vc7a_2b_3958628
schedule-build:396.vb_db_07b_4d29cf
scm-api:631.v9143df5b_e4a_a
script-security:1228.vd93135a_2fb_25
scriptler:3.5
sectioned-view:1.25
shiningpanda:0.24
sidebar-link:1.12.1
simple-parameterized-builds-report:1.5
simple-theme-plugin:136.v23a_15f86c53d
snakeyaml-api:1.33-90.v80dcb_3814d35
ssh:2.6.1
ssh-agent:295.v9ca_a_1c7cc3a_a_
ssh-credentials:305.v8f4381501156
ssh-slaves:2.854.v7fd446b_337c9
sshd:3.275.v9e17c10f2571
structs:324.va_f5d6774f3a_d
subversion:2.17.0
text-finder:1.22
thinBackup:1.15
throttle-concurrents:2.10
timestamper:1.21
token-macro:321.vd7cc1f2a_52c8
trilead-api:2.84.v72119de229b_7
uno-choice:2.5.7
validating-string-parameter:2.8
variant:59.vf075fe829ccb
view-job-filters:2.3
windows-slaves:1.8.1
workflow-aggregator:590.v6a_d052e5a_a_b_5
workflow-api:1200.v8005c684b_a_c6
workflow-basic-steps:994.vd57e3ca_46d24
workflow-cps:3601.v9b_36a_d99e1cc
workflow-cps-global-lib:609.vd95673f149b_b
workflow-durable-task-step:1217.v38306d8fa_b_5c
workflow-job:1254.v3f64639b_11dd
workflow-multibranch:716.vc692a_e52371b_
workflow-scm-step:400.v6b_89a_1317c9a_
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:839.v35e2736cfd5c
ws-cleanup:0.44
xvnc:1.28
zentimestamp:4.2

2

parts of the pipeline run on the controller node regardless of what you’ve specified in your Jenkinsfile. It may be worth looking into what the actual pipeline is here.

3

I have similar question, but not quite same. I use kubernetes as the cloud provider, and i can always see some pipeline jobs being “executed” on built-in node, but after a couple hundred milliseconds they’ll be “delivered” to nodes (in other words: pods) during the busy time in workdays.

  1. The pipeline job is delegated to built-in node
  2. Then the new node(pod) will take this pipeline job
    image

Is it the same problem that built-in node is running parts of pipeline script which are not control-able for us?

4

The controller runs the Jenkinsfile itself because part of the pipeline identifies what agents to use. Once the pipeline has specified a specific node and that node is available, then the remainder of the job is processed on the agent itself.

5

we have the same issue on our Jenkins - 2.414.3 , when the load is high - more than 200 agents connected + aws instances + k8s pods , I can see that the controller holds a list of executers , although
the configuration on it is 0 executers - no pipeline should run on it …

Kevin , any chance you find a fix for it ?

6

In my experience, most memory issues boil down to not cleaning up old jobs ( Jenkins really tries to read all job results it can find into memory). So make sure each jobs configures discarding old jobs in a sane way.

7

I would say that over 90% of pipeline code runs on the controller and only when it comes to executing things on the agent then something happens on the agent.
Lets look at a simple pipeline (scripted)

node('myagent') {
  echo "start of node"
  dir('test') {
    echo "inside test directory"
    withEnv(['myvar=value']) {
      sh '''
        echo $myvar
      '''
    }
  }
}

the first time, that code is running on the agent is the sh step, the node step alone is not yet creating the workspace and also the dir step isn’t creating the workspace.
Only the sh step will create a temporary file on the agent containing the script and then it will call the script. There is some code now running decoupled from the agent process (to ensure that a restart doesn’t kill the process), and some code inside the agent process that will forward log output to the controller and reap the status once the sh step finished.
All the echo steps are run on the controller. It would only be unnecessary overhead to first transfer something to the agent java process just to print something to the log.

This is not different from freestyle jobs where also most code is running in the controller and only when it comes to actually executing an external process or collecting files from the workspace that there is code executed on the agent.

The difference between freestyle and pipeline as that a freestyle job always consumes an executor while a pipeline job only while inside a node step.
pipeline jobs create flyweight executors on the controller that sometimes are listed on the controller but those do not consume any of the controllers explicit executors and appear without the executor number. This is normal and expected behaviour.