Starting Jenkins failed - Process /usr/bin/jenkins could not be executed

Using Jenkins Ask a question
1

When trying a clean install for jenkins (jenkins-2.370-1.1.noarch.rpm), but we ran into an issue
As far as we can tell all permissions should be ok
What are we missing, we been trying for days now ?

-- Unit jenkins.service has begun starting up.
<redacted> systemd[2644]: jenkins.service: Changing to the requested working directory failed: Permission denied
<redacted>  systemd[2644]: jenkins.service: Failed at step CHDIR spawning /usr/bin/jenkins: Permission denied
-- Subject: Process /usr/bin/jenkins could not be executed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- The process /usr/bin/jenkins could not be executed and failed.
--
-- The error number returned by this process is 13.

Here is the systemctl -l status jenkins log

<redacted> [/tmp/install]: sudo systemctl -l status jenkins
● jenkins.service - Jenkins Continuous Integration Server
   Loaded: loaded (/usr/lib/systemd/system/jenkins.service; disabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/jenkins.service.d
           └─override.conf
   Active: failed (Result: exit-code) since Mon 2022-09-26 15:25:14 CEST; 10min ago
  Process: 2644 ExecStart=/usr/bin/jenkins (code=exited, status=200/CHDIR)
 Main PID: 2644 (code=exited, status=200/CHDIR)

<redacted> systemd[1]: jenkins.service: Main process exited, code=exited, status=200/CHDIR
<redacted> systemd[1]: jenkins.service: Failed with result 'exit-code'.
<redacted> systemd[1]: Failed to start Jenkins Continuous Integration Server.
<redacted> systemd[1]: jenkins.service: Service RestartSec=100ms expired, scheduling restart.
<redacted> systemd[1]: jenkins.service: Scheduled restart job, restart counter is at 5.
<redacted> systemd[1]: Stopped Jenkins Continuous Integration Server.
<redacted> systemd[1]: jenkins.service: Start request repeated too quickly.
<redacted> systemd[1]: jenkins.service: Failed with result 'exit-code'.
<redacted> systemd[1]: Failed to start Jenkins Continuous Integration Server.
2

Check the permissions of the /var/lib/jenkins directory. You state that the permissions should be ok, but that seems the most likely source of the problem. Permissions of one of the parent directories might also be an issue.

If you have enabled one of the Red Hat security profiles, you may need to reconsider that choice.

3

The permissions
Note the jenkins user permissions where put in manually to see if it helped (it didnt)

# file: var/lib/jenkins
# owner: jenkins
# group: jenkins
user::rwx
group::r-x
other::r-x

# file: var/lib
# owner: root
# group: root
user::rwx
group::r-x
other::r-x

# file: var
# owner: root
# group: root
user::rwx
group::r-x
other::r-x

# file: usr/bin/jenkins
# owner: jenkins
# group: jenkins
user::rwx
group::r-x
other::r-x

# file: usr/bin
# owner: root
# group: root
user::rwx
user:jenkins:rwx
group::r-x
mask::rwx
other::r-x

# file: usr
# owner: root
# group: root
user::rwx
user:jenkins:rwx
group::r-x
mask::rwx
other::r-x
4

Any chance you set ProtectHome=true?

Or possibly, that you’re using a network file system and encountering a systemd bug as described in

5

systemctl show jenkins says ProtectHome=no

systemd version systemctl --version says
systemd 239 (239-58.el8_6.7)

Additional info

  Operating System: Red Hat Enterprise Linux 8.6 (Ootpa)
       CPE OS Name: cpe:/o:redhat:enterprise_linux:8::baseos
            Kernel: Linux 4.18.0-372.26.1.el8_6.x86_64
      Architecture: x86-64
6

Interesting, since that is the exact operating system where I have been running a Jenkins 2.361.1 controller. I don’t have other suggestions to offer. We test variants of that operating system with test automation (Rocky 9, Alma 9, Oracle Linux 9) and I use Red Hat Enterprise Linux 8.6 interactively.

7

I willl see if i somehow can update systemd, just to be able to rule it out

8

The version of systemd that you are running matches exactly with the systemd on my RHEL 8.6 system. No harm trying to update it, but I suspect something else in the system configuration, not the systemd version.

9

what does your override have?

What filesystem is /var/lib/jenkins (nfs?)

10

We found the cause
sudo cat /var/log/audit/audit.log | grep jenkins | grep denied gave result

type=AVC msg=audit(1664206374.094:405): avc:  denied  { read } for  pid=2849 comm="(jenkins)" name="jenkins" dev="dm-6" ino=                                                                                  2098029 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=lnk_file permissive=0

Somewhere in the installation script we used, there was a move from /var/lib/jenkins to /srv/jenkins
This was done by creating a symlink to /srv/jenkins
Why this was ever in the scripts we are still looking into that
But it seems that SELinux does not permit the use of symlink in this instance (if i understand correctly)

So be removing this ‘move’, we now have a working Jenkins instance :smiley:

11

I havn’t tried it, but you can probably set the JENKINS_HOME env variable in overrides to point to a different location, no need for symlink.

12

We will do that, once we know why there was a move/symlink in the first place

Thanks for all the help