"Cannot run program "nohup"" on RedHat Jenkins Agents

Yesterday (May 6), one of my agents failed a build due to, “Cannot run program “nohup””:

java.io.IOException: Cannot run program "nohup" (in directory "/var/lib/jenkins/workspace/tinker"): error=2, No such file or directory

Today, I am digging in, and it is immediately apparent that this issue is occurring on ALL of my agents:

  • When a sh step is called: Jenkins fails to find “nohup”

:information_source: My agents are running RedHat Enterprise Linux 8 and 9.

Possible cause (or red herring fallacy): I did just update from Java 11 to Java 21 on May 2, 2024.

Note: Three of my nightly builds ran fine using Java 21 (on May 3rd, 4th, and 5th, at midnight PT).
Finally, on May 6th, it began failing.

I will try out using Java 11 again and report back.

Thanks for any help!

Jenkins setup:
Jenkins: 2.452
OS: Linux - 4.18.0-513.9.1.el8_9.x86_64
Java: 21.0.2 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)

active-directory:2.35
ansicolor:1.0.4
ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
asm-api:9.7-33.v4d23ef79fcc8
authentication-tokens:1.53.v1c90fd9191a_b_
basic-branch-build-strategies:81.v05e333931c7d
blueocean:1.27.11
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.11
blueocean-commons:1.27.11
blueocean-config:1.27.11
blueocean-core-js:1.27.11
blueocean-dashboard:1.27.11
blueocean-display-url:2.4.2
blueocean-events:1.27.11
blueocean-git-pipeline:1.27.11
blueocean-github-pipeline:1.27.11
blueocean-i18n:1.27.11
blueocean-jwt:1.27.11
blueocean-personalization:1.27.11
blueocean-pipeline-api-impl:1.27.11
blueocean-pipeline-editor:1.27.11
blueocean-pipeline-scm-api:1.27.11
blueocean-rest:1.27.11
blueocean-rest-impl:1.27.11
blueocean-web:1.27.11
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9
branch-api:2.1152.v6f101e97dd77
build-timeout:1.32
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.0.2
cloudbees-bitbucket-branch-source:883.v041fa_695e9c2
cloudbees-folder:6.901.vb_4c7a_da_75da_3
command-launcher:107.v773860566e2e
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.11.0-95.v22a_d30ee5d36
configuration-as-code:1775.v810dc950b_514
coverage:1.13.0
credentials:1337.v60b_d7b_c7b_c9f
credentials-binding:657.v2b_19db_7d6e6d
data-tables-api:2.0.2-1
display-url-api:2.200.vb_9327d658781
docker-commons:439.va_3cb_0a_6a_fb_29
docker-workflow:572.v950f58993843
durable-task:550.v0930093c4b_a_6
echarts-api:5.5.0-1
email-ext:2.105
external-monitor-job:215.v2e88e894db_f8
favorite:2.208.v91d65b_7792a_c
font-awesome-api:6.5.1-3
forensics-api:2.4.0
git:5.2.1
git-client:4.7.0
git-server:114.v068a_c7cc2574
github:1.38.0
github-api:1.318-461.v7a_c09c9fa_d63
github-branch-source:1785.v99802b_69816c
gradle:2.10
gson-api:2.10.1-15.v0d99f670e0a_7
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
htmlpublisher:1.33
instance-identity:185.v303dc7c645f9
ionicons-api:70.v2959a_b_74e3cf
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jenkins-design-language:1.27.11
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
jnr-posix-api:3.1.19-2
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery3-api:3.7.1-2
jsch:0.2.16-86.v42e010d9484b_
json-api:20240303-41.v94e11e6de726
json-path-api:2.9.0-58.v62e3e85b_a_655
junit:1265.v65b_14fa_f12f0
ldap:719.vcb_d039b_77d0d
lockable-resources:1246.v28b_e4cc6fa_16
mailer:472.vf7c289a_4b_420
markdown-formatter:167.v8a_428ca_49f89
matrix-auth:3.2.2
matrix-project:822.824.v14451b_c0fd42
mina-sshd-api-common:2.12.1-101.v85b_e08b_780dd
mina-sshd-api-core:2.12.1-101.v85b_e08b_780dd
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
pam-auth:1.10
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-groovy-lib:704.vc58b_8890a_384
pipeline-input-step:491.vb_07d21da_1a_fb_
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2184.v0b_358b_953e69
pipeline-model-definition:2.2184.v0b_358b_953e69
pipeline-model-extensions:2.2184.v0b_358b_953e69
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2184.v0b_358b_953e69
pipeline-stage-view:2.34
plain-credentials:179.vc5cb_98f6db_38
plugin-util-api:4.1.0
prism-api:1.29.0-13
pubsub-light:1.18
pyenv-pipeline:2.1.2
resource-disposer:0.23
scm-api:689.v237b_6d3a_ef7f
script-security:1326.vdb_c154de8669
snakeyaml-api:2.2-111.vc6598e30cc65
sse-gateway:1.26
ssh-agent:346.vda_a_c4f2c8e50
ssh-credentials:334.v7732563deee1
ssh-slaves:2.948.vb_8050d697fec
sshd:3.322.v159e91f6a_550
structs:337.v1b_04ea_4df7c8
thinBackup:1.19
timestamper:1.26
token-macro:400.v35420b_922dcb_
trilead-api:2.142.v748523a_76693
variant:60.v7290fc0eb_b_cd
workflow-aggregator:596.v8c21c963d92d
workflow-api:1291.v51fd2a_625da_7
workflow-basic-steps:1049.v257a_e6b_30fb_d
workflow-cps:3894.vd0f0248b_a_fc4
workflow-durable-task-step:1336.v768003e07199
workflow-job:1400.v7fd111b_ec82f
workflow-multibranch:773.vc4fe1378f1d5
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:657.v03b_e8115821b_
workflow-support:896.v175a_a_9c5b_78f
ws-cleanup:0.45

It does look like Java 21 is the issue – changing back to Java 11 fixed the issue!

(WORKING) Jenkins setup:
Jenkins: 2.452
OS: Linux - 4.18.0-513.9.1.el8_9.x86_64
Java: 11.0.23 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)

active-directory:2.35
ansicolor:1.0.4
ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
asm-api:9.7-33.v4d23ef79fcc8
authentication-tokens:1.53.v1c90fd9191a_b_
basic-branch-build-strategies:81.v05e333931c7d
blueocean:1.27.11
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.11
blueocean-commons:1.27.11
blueocean-config:1.27.11
blueocean-core-js:1.27.11
blueocean-dashboard:1.27.11
blueocean-display-url:2.4.2
blueocean-events:1.27.11
blueocean-git-pipeline:1.27.11
blueocean-github-pipeline:1.27.11
blueocean-i18n:1.27.11
blueocean-jwt:1.27.11
blueocean-personalization:1.27.11
blueocean-pipeline-api-impl:1.27.11
blueocean-pipeline-editor:1.27.11
blueocean-pipeline-scm-api:1.27.11
blueocean-rest:1.27.11
blueocean-rest-impl:1.27.11
blueocean-web:1.27.11
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9
branch-api:2.1152.v6f101e97dd77
build-timeout:1.32
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.0.2
cloudbees-bitbucket-branch-source:883.v041fa_695e9c2
cloudbees-folder:6.901.vb_4c7a_da_75da_3
command-launcher:107.v773860566e2e
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.11.0-95.v22a_d30ee5d36
configuration-as-code:1775.v810dc950b_514
coverage:1.13.0
credentials:1337.v60b_d7b_c7b_c9f
credentials-binding:657.v2b_19db_7d6e6d
data-tables-api:2.0.2-1
display-url-api:2.200.vb_9327d658781
docker-commons:439.va_3cb_0a_6a_fb_29
docker-workflow:572.v950f58993843
durable-task:550.v0930093c4b_a_6
echarts-api:5.5.0-1
email-ext:2.105
external-monitor-job:215.v2e88e894db_f8
favorite:2.208.v91d65b_7792a_c
font-awesome-api:6.5.1-3
forensics-api:2.4.0
git:5.2.1
git-client:4.7.0
git-server:114.v068a_c7cc2574
github:1.38.0
github-api:1.318-461.v7a_c09c9fa_d63
github-branch-source:1785.v99802b_69816c
gradle:2.10
gson-api:2.10.1-15.v0d99f670e0a_7
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
htmlpublisher:1.33
instance-identity:185.v303dc7c645f9
ionicons-api:70.v2959a_b_74e3cf
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jenkins-design-language:1.27.11
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
jnr-posix-api:3.1.19-2
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery3-api:3.7.1-2
jsch:0.2.16-86.v42e010d9484b_
json-api:20240303-41.v94e11e6de726
json-path-api:2.9.0-58.v62e3e85b_a_655
junit:1265.v65b_14fa_f12f0
ldap:719.vcb_d039b_77d0d
lockable-resources:1246.v28b_e4cc6fa_16
mailer:472.vf7c289a_4b_420
markdown-formatter:167.v8a_428ca_49f89
matrix-auth:3.2.2
matrix-project:822.824.v14451b_c0fd42
mina-sshd-api-common:2.12.1-101.v85b_e08b_780dd
mina-sshd-api-core:2.12.1-101.v85b_e08b_780dd
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
pam-auth:1.10
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-groovy-lib:704.vc58b_8890a_384
pipeline-input-step:491.vb_07d21da_1a_fb_
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2184.v0b_358b_953e69
pipeline-model-definition:2.2184.v0b_358b_953e69
pipeline-model-extensions:2.2184.v0b_358b_953e69
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2184.v0b_358b_953e69
pipeline-stage-view:2.34
plain-credentials:179.vc5cb_98f6db_38
plugin-util-api:4.1.0
prism-api:1.29.0-13
pubsub-light:1.18
pyenv-pipeline:2.1.2
resource-disposer:0.23
scm-api:689.v237b_6d3a_ef7f
script-security:1326.vdb_c154de8669
snakeyaml-api:2.2-111.vc6598e30cc65
sse-gateway:1.26
ssh-agent:346.vda_a_c4f2c8e50
ssh-credentials:334.v7732563deee1
ssh-slaves:2.948.vb_8050d697fec
sshd:3.322.v159e91f6a_550
structs:337.v1b_04ea_4df7c8
thinBackup:1.19
timestamper:1.26
token-macro:400.v35420b_922dcb_
trilead-api:2.142.v748523a_76693
variant:60.v7290fc0eb_b_cd
workflow-aggregator:596.v8c21c963d92d
workflow-api:1291.v51fd2a_625da_7
workflow-basic-steps:1049.v257a_e6b_30fb_d
workflow-cps:3894.vd0f0248b_a_fc4
workflow-durable-task-step:1336.v768003e07199
workflow-job:1400.v7fd111b_ec82f
workflow-multibranch:773.vc4fe1378f1d5
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:657.v03b_e8115821b_
workflow-support:896.v175a_a_9c5b_78f
ws-cleanup:0.45

Welcome to this community, @ryeleo, and thanks for the feedback. :wave:

Now, the question is: why are you facing this issue with Java 21? :thinking:
Can you run your tests with a Temurin JDK21?

This happens when you run a sh step?
You might want to set
org.jenkinsci.plugins.durabletask.BourneShellScript.USE_BINARY_WRAPPER=true via script console or permanently as a java property in the java command that starts jenkins (java -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.USE_BINARY_WRAPPER=true ... -jar jenkins.war ...)

1 Like

We also use RHEL8.9 agents with JDK21. We’ve been using this setup without issue for approximately 2 months. We’re on Jenkins 2.463 and have upgraded a couple of times in the last few months. We just started seeing the same error about nohup, but it only appears intermittently. Some builds run without issue, others throw an exception when using sh.

Is there any additional information on this issue?

edit: downgrading RHEL8 agents to JDK17 appears to have fixed the issue for us.

Some additional info, after downgrading to JDK 17 the issue was resolved for approximately one day and then later resurfaced. We use GCP VMs running RHEL8 as Jenkins agents. These are created by Jenkins from an image. The image had JDK version 17.0.11 installed. Jenkins picks up this version of the JDK. dnf-automatic was installed on the VMs, and after about 5 minutes of uptime dnf-automatic was upgrading the JDK version to 17.0.12, but Jenkins already read the version as 17.0.11. Removing dnf-automatic so the JDK version remains at 17.0.11 appears to have fixed the problem

Hi, same here, rhel8 and rhel9.
With JDK 17.0.12 all my pipelines break with nohup not being able to run, while 17.0.11 works fine.
Is that more a java or a jenkins issue? Or even a security policy thingy?

Thanks for the advice! This does maybe seem to get me past the ‘nohup’ issue, but I am now running into similar issues:

Contxt: I re-upgraded to 21 around July 2nd. For a month, all seemed well. However, yesterday, I am starting to see issues with Java 21 not finding executables again! Specifically, this morning I am seeing that Jenkins is unable to find the following executables so far:

  • nohup
  • cp
  • python3.11

I was surprised that org.jenkinsci.plugins.durabletask.BourneShellScript.USE_BINARY_WRAPPER=true did seem to get me past the nohup issue, but now I am running into issues with cp and python3.11 being unavailable (but they certainly are available. I’ve confirmed the jenkins user can indeed run cp and python3.11 on my agents.)

Do you have by chance automatic updates enabled for java in your package manager and java was updated but your agents were not restarted?
See 2 comments above from @ciaconet

There is a chance that it has something to do with java being updated automatically via our patching process. I can look more into this.

I will try restarting my Jenkins Controller entirely, which I believe should have the effect of restarting all of the agents as well.

Note: I am debugging this and it seems that even the most simple pipeline is failing to run due to:
Caused: java.io.IOException: Cannot run program "/var/lib/jenkins/caches/durable-task/durable_task_monitor_555.v6802fe0f0b_82_linux_64" (in directory "/var/lib/jenkins/workspace/tinker"): error=2, No such file or directory"

Restarting my Jenkins Controller using does seem to have fixed my issues this morning for most of my pipelines. :dizzy_face:

Per @mawinter69, I checked if Java 21 was recently updated. It does indeed seem that it was which does correspond to the issues I’ve been seeing most recently!

$ yum history | head
Not root, Subscription Management repositories not updated
ID  | Command line                          | Date and time    | Action(s)      | Altered
------------------------------------------------------------------------------------------------
... trimmed for brevity...
142 | -d 0 -e 1 -y upgrade java-21-openjdk  | 2024-08-06 02:02 | Upgrade        |    2   
... trimmed for brevity...