CMake running extremely slowly on macOS Ventura and Sonoma nodes

Using Jenkins Ask a question
1

We’ve been successfully using Jenkins with macOS Monterrey x86_64 build nodes for some time. We’re trying to set up some macOS Ventura and Sonoma build nodes that use M1 processors and experiencing a very frustrating problem.

We use groovy pipelines in Jenkins for overall orchestration. Our pipeline clones a few git repos to begin with, and that works fine. Then it uses CMake (run via a Python script, which runs a ctest script) to configure. We’re primarily a C++ and C code.

The configure step takes forever. In contrast to our Monterrey machines, where configuration takes about 1.5 minutes, it takes hours on the Ventura and Sonoma ones. (I’ve actually never waited around for it to complete. The job times out after 2 hours.) It’s not getting stuck on any particular step. It’s just proceeding very, very slowly.

If I ssh into the build node and do the configuration “by hand” by running the same scripts, it works normally.

Things we’ve tried so far:

  • Rebooting the nodes and Jenkins
  • We are using arm64 openjdk for our Java. We’ve tried a couple of different versions, 17.0.2 and 21.0.2.
  • Moving the workspace to a different part of the file system.
  • Watching top while a build runs to see if any other processes are consuming a lot of system resources (nothing more than a few percent of CPU)
  • Connecting the nodes two ways: via ssh and “launch agent by connecting it to the controller”
  • Updating our plugins
  • Different users
  • Comparing the environment, at the time CMake is run, of the working Monterrey build to the Ventura/Sonoma builds (nothing sticks out as weird.)

I would greatly appreciate any hints about things to look at.

Jenkins setup:

Jenkins: 2.426.3
OS: Linux - 4.18.0-372.80.1.el8_6.x86_64
Java: 11.0.22 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)

ansicolor:1.0.4
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
apache-httpcomponents-client-5-api:5.3.1-1.0
authentication-tokens:1.53.v1c90fd9191a_b_
blueocean:1.27.10
blueocean-bitbucket-pipeline:1.27.10
blueocean-commons:1.27.10
blueocean-config:1.27.10
blueocean-core-js:1.27.10
blueocean-dashboard:1.27.10
blueocean-display-url:2.4.2
blueocean-events:1.27.10
blueocean-git-pipeline:1.27.10
blueocean-github-pipeline:1.27.10
blueocean-i18n:1.27.10
blueocean-jwt:1.27.10
blueocean-personalization:1.27.10
blueocean-pipeline-api-impl:1.27.10
blueocean-pipeline-editor:1.27.10
blueocean-pipeline-scm-api:1.27.10
blueocean-rest:1.27.10
blueocean-rest-impl:1.27.10
blueocean-web:1.27.10
bootstrap5-api:5.3.2-3
bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9
branch-api:2.1148.vce12cfcdf090
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.0.2
cloud-stats:336.v788e4055508b_
cloudbees-bitbucket-branch-source:873.v0ed259216f8d
cloudbees-disk-usage-simple:203.v3f46a_7462b_1a_
cloudbees-folder:6.858.v898218f3609d
command-launcher:107.v773860566e2e
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.11.0-95.v22a_d30ee5d36
config-file-provider:968.ve1ca_eb_913f8c
configuration-as-code:1775.v810dc950b_514
copyartifact:722.v0662a_9b_e22a_c
credentials:1319.v7eb_51b_3a_c97b_
credentials-binding:657.v2b_19db_7d6e6d
display-url-api:2.200.vb_9327d658781
docker-build-step:2.11
docker-commons:439.va_3cb_0a_6a_fb_29
docker-java-api:3.3.4-86.v39b_a_5ede342c
docker-plugin:1.5
docker-workflow:572.v950f58993843
durable-task:550.v0930093c4b_a_6
echarts-api:5.4.3-2
email-ext:2.104
favorite:2.208.v91d65b_7792a_c
font-awesome-api:6.5.1-2
git:5.2.1
git-client:4.6.0
github:1.38.0
github-api:1.318-461.v7a_c09c9fa_d63
github-branch-source:1772.va_69eda_d018d4
gitlab-plugin:1.8.0
gradle:2.10
gson-api:2.10.1-15.v0d99f670e0a_7
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
htmlpublisher:1.32
instance-identity:185.v303dc7c645f9
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.16.1-373.ve709c6871598
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javadoc:243.vb_b_503b_b_45537
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jenkins-design-language:1.27.10
jersey2-api:2.41-133.va_03323b_a_1396
jjwt-api:0.11.5-77.v646c772fddb_0
jobConfigHistory:1229.v3039470161a_d
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery3-api:3.7.1-1
jsch:0.2.16-86.v42e010d9484b_
json-api:20240205-27.va_007549e895c
json-path-api:2.9.0-33.v2527142f2e1d
junit:1259.v65ffcef24a_88
kubernetes:3971.v94b_4c914ca_75
kubernetes-client-api:6.4.1-215.v2ed17097a_8e9
kubernetes-credentials:0.10.0
ldap:711.vb_d1a_491714dc
mailer:463.vedf8358e006b_
markdown-formatter:95.v17a_965e696ee
matrix-auth:3.1.9
matrix-project:822.824.v14451b_c0fd42
mattermost:3.1.3
maven-plugin:3.23
metrics:4.2.21-449.v6960d7c54c69
mina-sshd-api-common:2.12.0-90.v9f7fb_9fa_3d3b_
mina-sshd-api-core:2.12.0-90.v9f7fb_9fa_3d3b_
nodejs:1.6.1
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
parameter-separator:166.vd0120849b_386
persistent-parameter:1.3
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:704.vc58b_8890a_384
pipeline-input-step:477.v339683a_8d55e
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2175.v76a_fff0a_2618
pipeline-model-definition:2.2175.v76a_fff0a_2618
pipeline-model-extensions:2.2175.v76a_fff0a_2618
pipeline-rest-api:2.34
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2175.v76a_fff0a_2618
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:3.8.0
prism-api:1.29.0-8
prometheus:2.5.1
publish-over:0.22
publish-over-ssh:1.25
pubsub-light:1.18
python:1.3
rebuild:330.v645b_7df10e2a_
resource-disposer:0.23
scm-api:683.vb_16722fb_b_80b_
script-security:1326.vdb_c154de8669
snakeyaml-api:2.2-111.vc6598e30cc65
sse-gateway:1.26
ssh-credentials:308.ve4497b_ccd8f4
ssh-slaves:2.948.vb_8050d697fec
sshd:3.322.v159e91f6a_550
structs:337.v1b_04ea_4df7c8
timestamper:1.26
token-macro:400.v35420b_922dcb_
trilead-api:2.133.vfb_8a_7b_9c5dd1
validating-string-parameter:183.v3748e79b_9737
variant:60.v7290fc0eb_b_cd
windows-slaves:1.8.1
workflow-aggregator:596.v8c21c963d92d
workflow-api:1291.v51fd2a_625da_7
workflow-basic-steps:1042.ve7b_140c4a_e0c
workflow-cps:3853.vb_a_490d892963
workflow-durable-task-step:1331.vc8c2fed35334
workflow-job:1385.vb_58b_86ea_fff1
workflow-multibranch:773.vc4fe1378f1d5
workflow-scm-step:415.v434365564324
workflow-step-api:657.v03b_e8115821b_
workflow-support:865.v43e78cc44e0d
ws-cleanup:0.45

2

Hello @jadamstephens and welcome to this community. :wave:

I don’t have access to M1 Mac, so I don’t know how to help.
The only thing that bothers me is you’re using a more recent version of Java on the agents than on the controller. :thinking:

3

Thanks for your suggestion. That also stuck out to me as a possible issue as I was copying and pasting the Jenkins setup.

I initially used openjdk 17 and 21 because those are the downloads I could find for arm64. Updating Jenkins controller to use a newer Java is trickier. I’m working with that admin of the machine that runs the controller to see if we can do it.

I’ve tried two more things.

First, I downloaded an x64 binary of openjdk11 and configured Jenkins to launch it using ‘arch -x86_64’. Surprisingly, the job ran at normal speed the first time I started it, but never again. I have the time stamped log that proves it, so I know I didn’t imagine it. I’ve tried a lot of different things to re-create this fluke, but to not avail.

Second, I realized this morning that homebrew provides an arm64 version of openjdk11. I installed it and set up Jenkins to use it, but it didn’t help.

So, apart from that one weird time that things magically worked, there’s not much indication that it’s a Java version mismatch problem.

1 Like
4

The problem ultimately turned out to be cybersecurity software required by my company. We obtained an exception and removed it, and the nodes are performing normally now.