"Cannot run program "nohup"" on RedHat Jenkins Agents

Yesterday (May 6), one of my agents failed a build due to, “Cannot run program “nohup””:

java.io.IOException: Cannot run program "nohup" (in directory "/var/lib/jenkins/workspace/tinker"): error=2, No such file or directory

Today, I am digging in, and it is immediately apparent that this issue is occurring on ALL of my agents:

  • When a sh step is called: Jenkins fails to find “nohup”

:information_source: My agents are running RedHat Enterprise Linux 8 and 9.

Possible cause (or red herring fallacy): I did just update from Java 11 to Java 21 on May 2, 2024.

Note: Three of my nightly builds ran fine using Java 21 (on May 3rd, 4th, and 5th, at midnight PT).
Finally, on May 6th, it began failing.

I will try out using Java 11 again and report back.

Thanks for any help!

Jenkins setup:
Jenkins: 2.452
OS: Linux - 4.18.0-513.9.1.el8_9.x86_64
Java: 21.0.2 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)

active-directory:2.35
ansicolor:1.0.4
ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
asm-api:9.7-33.v4d23ef79fcc8
authentication-tokens:1.53.v1c90fd9191a_b_
basic-branch-build-strategies:81.v05e333931c7d
blueocean:1.27.11
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.11
blueocean-commons:1.27.11
blueocean-config:1.27.11
blueocean-core-js:1.27.11
blueocean-dashboard:1.27.11
blueocean-display-url:2.4.2
blueocean-events:1.27.11
blueocean-git-pipeline:1.27.11
blueocean-github-pipeline:1.27.11
blueocean-i18n:1.27.11
blueocean-jwt:1.27.11
blueocean-personalization:1.27.11
blueocean-pipeline-api-impl:1.27.11
blueocean-pipeline-editor:1.27.11
blueocean-pipeline-scm-api:1.27.11
blueocean-rest:1.27.11
blueocean-rest-impl:1.27.11
blueocean-web:1.27.11
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9
branch-api:2.1152.v6f101e97dd77
build-timeout:1.32
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.0.2
cloudbees-bitbucket-branch-source:883.v041fa_695e9c2
cloudbees-folder:6.901.vb_4c7a_da_75da_3
command-launcher:107.v773860566e2e
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.11.0-95.v22a_d30ee5d36
configuration-as-code:1775.v810dc950b_514
coverage:1.13.0
credentials:1337.v60b_d7b_c7b_c9f
credentials-binding:657.v2b_19db_7d6e6d
data-tables-api:2.0.2-1
display-url-api:2.200.vb_9327d658781
docker-commons:439.va_3cb_0a_6a_fb_29
docker-workflow:572.v950f58993843
durable-task:550.v0930093c4b_a_6
echarts-api:5.5.0-1
email-ext:2.105
external-monitor-job:215.v2e88e894db_f8
favorite:2.208.v91d65b_7792a_c
font-awesome-api:6.5.1-3
forensics-api:2.4.0
git:5.2.1
git-client:4.7.0
git-server:114.v068a_c7cc2574
github:1.38.0
github-api:1.318-461.v7a_c09c9fa_d63
github-branch-source:1785.v99802b_69816c
gradle:2.10
gson-api:2.10.1-15.v0d99f670e0a_7
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
htmlpublisher:1.33
instance-identity:185.v303dc7c645f9
ionicons-api:70.v2959a_b_74e3cf
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jenkins-design-language:1.27.11
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
jnr-posix-api:3.1.19-2
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery3-api:3.7.1-2
jsch:0.2.16-86.v42e010d9484b_
json-api:20240303-41.v94e11e6de726
json-path-api:2.9.0-58.v62e3e85b_a_655
junit:1265.v65b_14fa_f12f0
ldap:719.vcb_d039b_77d0d
lockable-resources:1246.v28b_e4cc6fa_16
mailer:472.vf7c289a_4b_420
markdown-formatter:167.v8a_428ca_49f89
matrix-auth:3.2.2
matrix-project:822.824.v14451b_c0fd42
mina-sshd-api-common:2.12.1-101.v85b_e08b_780dd
mina-sshd-api-core:2.12.1-101.v85b_e08b_780dd
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
pam-auth:1.10
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-groovy-lib:704.vc58b_8890a_384
pipeline-input-step:491.vb_07d21da_1a_fb_
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2184.v0b_358b_953e69
pipeline-model-definition:2.2184.v0b_358b_953e69
pipeline-model-extensions:2.2184.v0b_358b_953e69
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2184.v0b_358b_953e69
pipeline-stage-view:2.34
plain-credentials:179.vc5cb_98f6db_38
plugin-util-api:4.1.0
prism-api:1.29.0-13
pubsub-light:1.18
pyenv-pipeline:2.1.2
resource-disposer:0.23
scm-api:689.v237b_6d3a_ef7f
script-security:1326.vdb_c154de8669
snakeyaml-api:2.2-111.vc6598e30cc65
sse-gateway:1.26
ssh-agent:346.vda_a_c4f2c8e50
ssh-credentials:334.v7732563deee1
ssh-slaves:2.948.vb_8050d697fec
sshd:3.322.v159e91f6a_550
structs:337.v1b_04ea_4df7c8
thinBackup:1.19
timestamper:1.26
token-macro:400.v35420b_922dcb_
trilead-api:2.142.v748523a_76693
variant:60.v7290fc0eb_b_cd
workflow-aggregator:596.v8c21c963d92d
workflow-api:1291.v51fd2a_625da_7
workflow-basic-steps:1049.v257a_e6b_30fb_d
workflow-cps:3894.vd0f0248b_a_fc4
workflow-durable-task-step:1336.v768003e07199
workflow-job:1400.v7fd111b_ec82f
workflow-multibranch:773.vc4fe1378f1d5
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:657.v03b_e8115821b_
workflow-support:896.v175a_a_9c5b_78f
ws-cleanup:0.45

It does look like Java 21 is the issue – changing back to Java 11 fixed the issue!

(WORKING) Jenkins setup:
Jenkins: 2.452
OS: Linux - 4.18.0-513.9.1.el8_9.x86_64
Java: 11.0.23 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)

active-directory:2.35
ansicolor:1.0.4
ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
asm-api:9.7-33.v4d23ef79fcc8
authentication-tokens:1.53.v1c90fd9191a_b_
basic-branch-build-strategies:81.v05e333931c7d
blueocean:1.27.11
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.11
blueocean-commons:1.27.11
blueocean-config:1.27.11
blueocean-core-js:1.27.11
blueocean-dashboard:1.27.11
blueocean-display-url:2.4.2
blueocean-events:1.27.11
blueocean-git-pipeline:1.27.11
blueocean-github-pipeline:1.27.11
blueocean-i18n:1.27.11
blueocean-jwt:1.27.11
blueocean-personalization:1.27.11
blueocean-pipeline-api-impl:1.27.11
blueocean-pipeline-editor:1.27.11
blueocean-pipeline-scm-api:1.27.11
blueocean-rest:1.27.11
blueocean-rest-impl:1.27.11
blueocean-web:1.27.11
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9
branch-api:2.1152.v6f101e97dd77
build-timeout:1.32
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.0.2
cloudbees-bitbucket-branch-source:883.v041fa_695e9c2
cloudbees-folder:6.901.vb_4c7a_da_75da_3
command-launcher:107.v773860566e2e
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.11.0-95.v22a_d30ee5d36
configuration-as-code:1775.v810dc950b_514
coverage:1.13.0
credentials:1337.v60b_d7b_c7b_c9f
credentials-binding:657.v2b_19db_7d6e6d
data-tables-api:2.0.2-1
display-url-api:2.200.vb_9327d658781
docker-commons:439.va_3cb_0a_6a_fb_29
docker-workflow:572.v950f58993843
durable-task:550.v0930093c4b_a_6
echarts-api:5.5.0-1
email-ext:2.105
external-monitor-job:215.v2e88e894db_f8
favorite:2.208.v91d65b_7792a_c
font-awesome-api:6.5.1-3
forensics-api:2.4.0
git:5.2.1
git-client:4.7.0
git-server:114.v068a_c7cc2574
github:1.38.0
github-api:1.318-461.v7a_c09c9fa_d63
github-branch-source:1785.v99802b_69816c
gradle:2.10
gson-api:2.10.1-15.v0d99f670e0a_7
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
htmlpublisher:1.33
instance-identity:185.v303dc7c645f9
ionicons-api:70.v2959a_b_74e3cf
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jenkins-design-language:1.27.11
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
jnr-posix-api:3.1.19-2
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery3-api:3.7.1-2
jsch:0.2.16-86.v42e010d9484b_
json-api:20240303-41.v94e11e6de726
json-path-api:2.9.0-58.v62e3e85b_a_655
junit:1265.v65b_14fa_f12f0
ldap:719.vcb_d039b_77d0d
lockable-resources:1246.v28b_e4cc6fa_16
mailer:472.vf7c289a_4b_420
markdown-formatter:167.v8a_428ca_49f89
matrix-auth:3.2.2
matrix-project:822.824.v14451b_c0fd42
mina-sshd-api-common:2.12.1-101.v85b_e08b_780dd
mina-sshd-api-core:2.12.1-101.v85b_e08b_780dd
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
pam-auth:1.10
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-groovy-lib:704.vc58b_8890a_384
pipeline-input-step:491.vb_07d21da_1a_fb_
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2184.v0b_358b_953e69
pipeline-model-definition:2.2184.v0b_358b_953e69
pipeline-model-extensions:2.2184.v0b_358b_953e69
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2184.v0b_358b_953e69
pipeline-stage-view:2.34
plain-credentials:179.vc5cb_98f6db_38
plugin-util-api:4.1.0
prism-api:1.29.0-13
pubsub-light:1.18
pyenv-pipeline:2.1.2
resource-disposer:0.23
scm-api:689.v237b_6d3a_ef7f
script-security:1326.vdb_c154de8669
snakeyaml-api:2.2-111.vc6598e30cc65
sse-gateway:1.26
ssh-agent:346.vda_a_c4f2c8e50
ssh-credentials:334.v7732563deee1
ssh-slaves:2.948.vb_8050d697fec
sshd:3.322.v159e91f6a_550
structs:337.v1b_04ea_4df7c8
thinBackup:1.19
timestamper:1.26
token-macro:400.v35420b_922dcb_
trilead-api:2.142.v748523a_76693
variant:60.v7290fc0eb_b_cd
workflow-aggregator:596.v8c21c963d92d
workflow-api:1291.v51fd2a_625da_7
workflow-basic-steps:1049.v257a_e6b_30fb_d
workflow-cps:3894.vd0f0248b_a_fc4
workflow-durable-task-step:1336.v768003e07199
workflow-job:1400.v7fd111b_ec82f
workflow-multibranch:773.vc4fe1378f1d5
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:657.v03b_e8115821b_
workflow-support:896.v175a_a_9c5b_78f
ws-cleanup:0.45

Welcome to this community, @ryeleo, and thanks for the feedback. :wave:

Now, the question is: why are you facing this issue with Java 21? :thinking:
Can you run your tests with a Temurin JDK21?

This happens when you run a sh step?
You might want to set
org.jenkinsci.plugins.durabletask.BourneShellScript.USE_BINARY_WRAPPER=true via script console or permanently as a java property in the java command that starts jenkins (java -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.USE_BINARY_WRAPPER=true ... -jar jenkins.war ...)