Systemd is failing to launch jenkins

ddas · November 21, 2024, 4:55pm

Systemd is failing to launch Jenkins with error message

jenkins.service: Job jenkins.service/start failed with result 'dependency'.

I tried launching jenkins with jenkins username, also tried a different username defining its name in override.conf but I am still getting same error. Please suggest.

Jenkins setup:

$ grep -vE '^$|^#' /etc/default/jenkins
NAME=jenkins
JAVA_ARGS="-Djava.awt.headless=true"
PIDFILE=/var/run/$NAME/$NAME.pid
JENKINS_USER=$NAME
JENKINS_GROUP=$NAME
JENKINS_WAR=/usr/share/java/$NAME.war
JENKINS_HOME=/var/lib/$NAME
RUN_STANDALONE=true
JENKINS_LOG=/var/log/$NAME/$NAME.log
JENKINS_ENABLE_ACCESS_LOG="no"
MAXOPENFILES=8192
HTTP_PORT=8080
PREFIX=/$NAME
JENKINS_ARGS="--webroot=/var/cache/$NAME/war --httpPort=$HTTP_PORT"

$ sudo systemctl status jenkins.service 
● jenkins.service - Jenkins Continuous Integration Server
     Loaded: loaded (/lib/systemd/system/jenkins.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/jenkins.service.d
             └─override.conf
     Active: inactive (dead)

Nov 21 08:01:54 ppe-jenkins-worker01 systemd[1]: Dependency failed for Jenkins Continuous Integration Server.
Nov 21 08:01:54 ppe-jenkins-worker01 systemd[1]: jenkins.service: Job jenkins.service/start failed with result 'dependency'.
Nov 21 08:11:12 ppe-jenkins-worker01 systemd[1]: Dependency failed for Jenkins Continuous Integration Server.
Nov 21 08:11:12 ppe-jenkins-worker01 systemd[1]: jenkins.service: Job jenkins.service/start failed with result 'dependency'.
Nov 21 08:19:04 ppe-jenkins-worker01 systemd[1]: Dependency failed for Jenkins Continuous Integration Server.
Nov 21 08:19:04 ppe-jenkins-worker01 systemd[1]: jenkins.service: Job jenkins.service/start failed with result 'dependency'.
Nov 21 16:36:48 ppe-jenkins-worker01 systemd[1]: Dependency failed for Jenkins Continuous Integration Server.
Nov 21 16:36:48 ppe-jenkins-worker01 systemd[1]: jenkins.service: Job jenkins.service/start failed with result 'dependency'.
Nov 21 16:47:08 ppe-jenkins-worker01 systemd[1]: Dependency failed for Jenkins Continuous Integration Server.
Nov 21 16:47:08 ppe-jenkins-worker01 systemd[1]: jenkins.service: Job jenkins.service/start failed with result 'dependency'.

$ ls -ld /var/lib/jenkins/
drwxr-xr-x 2 jenkins jenkins 4096 Oct  2 11:43 /var/lib/jenkins/

$ ls -ld /var/log/jenkins/
drwxr-xr-x 2 jenkins jenkins 4096 Oct  2 11:43 /var/log/jenkins/

$ ls -ld /var/cache/jenkins
drwxr-x--- 2 jenkins jenkins 4096 Oct  2 11:43 /var/cache/jenkins

Packages:

$ dpkg -l | grep jenkins
ii  jenkins                                2.462.3                           all          Jenkins is the leading open source automation server supported by a large and growing community of developers, testers, designers and other people interested in continuous integration, continuous delivery and modern software delivery practices. Built on the Java Virtual Machine (JVM), it provides more than 1,800 plugins that extend Jenkins to automate with practically any technology software delivery teams use. In 2022, Jenkins reached 300,000 known installations making it the most widely deployed automation server.
$ dpkg -l | grep java
ii  ca-certificates-java                   20190405ubuntu1.1                 all          Common CA certificates (JKS keystore)
ii  java-common                            0.72                              all          Base package for Java runtimes
ii  libatk-wrapper-java                    0.37.1-1                          all          ATK implementation for Java using JNI
ii  libatk-wrapper-java-jni:amd64          0.37.1-1                          amd64        ATK implementation for Java using JNI (JNI bindings)
$ dpkg -l | grep jdk
ii  openjdk-17-jre:amd64                   17.0.13+11-2ubuntu1~20.04         amd64        OpenJDK Java runtime, using Hotspot JIT
ii  openjdk-17-jre-headless:amd64          17.0.13+11-2ubuntu1~20.04         amd64        OpenJDK Java runtime, using Hotspot JIT (headless)

Content of override.conf

$ cat /etc/systemd/system/jenkins.service.d/override.conf
[Service]
Environment="JAVA_OPTS=-Xms4g -Xmx16g -XX:+UseG1GC -Dorg.jenkinsci.plugins.pipeline.modeldefinition.parser.RuntimeASTTransformer.SCRIPT_SPLITTING_TRANSFORMATION=false -Dhudson.model.DirectoryBrowserSupport.CSP="
Environment="JENKINS_LOG=/var/log/jenkins/jenkins.log"

poddingue · November 21, 2024, 7:28pm

We would need logs to try and understand what’s going on.

mawinter69 · November 21, 2024, 8:11pm

hmm, the Jenkins service only depends on the network service. So if that would be down and not starting Jenkins itself will not start.

MarkEWaite · November 21, 2024, 8:47pm

Only peripherally related to your question, but comments provided in case they help.

Jenkins does not use /etc/default/jenkins when installed with a Linux package manager. The systemd overrides are used to customize options.

Jenkins support for Ubuntu 20.04 will end in less than 6 months when Ubuntu ends their public support of Ubuntu 20.04. Use Ubuntu 22.04 or Ubuntu 24.04 for a longer operating system life and a longer time running on an operating system that Jenkins supports.

ddas · November 22, 2024, 3:56am

Thanks for looking. Unfortunately, I do not see any useful longlines other than these, initially I thought something systemd is depending on is failing but I am not sure what is it, network connection is good, can ping the gateway, can reach internet.

$ ping 8.8.8.8 -c 4
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=55 time=1.78 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=55 time=1.84 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=55 time=1.83 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=55 time=1.88 ms

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 1.778/1.832/1.876/0.035 ms

journalctl -u jenkins provides these below

Nov 21 07:42:20 ppe-jenkins-worker01 systemd[1]: Dependency failed for Jenkins Continuous Integration Server.
Nov 21 07:42:20 ppe-jenkins-worker01 systemd[1]: jenkins.service: Job jenkins.service/start failed with result 'dependency'.
Nov 21 07:43:53 ppe-jenkins-worker01 systemd[1]: Dependency failed for Jenkins Continuous Integration Server.
Nov 21 07:43:53 ppe-jenkins-worker01 systemd[1]: jenkins.service: Job jenkins.service/start failed with result 'dependency'.
-- Reboot --
Nov 21 08:01:54 ppe-jenkins-worker01 systemd[1]: Dependency failed for Jenkins Continuous Integration Server.
Nov 21 08:01:54 ppe-jenkins-worker01 systemd[1]: jenkins.service: Job jenkins.service/start failed with result 'dependency'.
Nov 21 08:11:12 ppe-jenkins-worker01 systemd[1]: Dependency failed for Jenkins Continuous Integration Server.
Nov 21 08:11:12 ppe-jenkins-worker01 systemd[1]: jenkins.service: Job jenkins.service/start failed with result 'dependency'.
Nov 21 08:19:04 ppe-jenkins-worker01 systemd[1]: Dependency failed for Jenkins Continuous Integration Server.
Nov 21 08:19:04 ppe-jenkins-worker01 systemd[1]: jenkins.service: Job jenkins.service/start failed with result 'dependency'.
Nov 21 16:36:48 ppe-jenkins-worker01 systemd[1]: Dependency failed for Jenkins Continuous Integration Server.
Nov 21 16:36:48 ppe-jenkins-worker01 systemd[1]: jenkins.service: Job jenkins.service/start failed with result 'dependency'.
Nov 21 16:47:08 ppe-jenkins-worker01 systemd[1]: Dependency failed for Jenkins Continuous Integration Server.
Nov 21 16:47:08 ppe-jenkins-worker01 systemd[1]: jenkins.service: Job jenkins.service/start failed with result 'dependency'.
-- Reboot --
Nov 21 17:01:10 ppe-jenkins-worker01 systemd[1]: Dependency failed for Jenkins Continuous Integration Server.
Nov 21 17:01:10 ppe-jenkins-worker01 systemd[1]: jenkins.service: Job jenkins.service/start failed with result 'dependency'.

ddas · November 22, 2024, 4:01am

Thanks for your response. I am able to reach the gateway and internet, so should not be network related, I could ssh to this jenkins box from a remote system in same network.

$ ping -c 4 pkg.jenkins.io
PING dualstack.d.sni.global.fastly.net (199.232.46.133) 56(84) bytes of data.
64 bytes from 199.232.46.133 (199.232.46.133): icmp_seq=1 ttl=51 time=36.7 ms
64 bytes from 199.232.46.133 (199.232.46.133): icmp_seq=2 ttl=51 time=37.0 ms
64 bytes from 199.232.46.133 (199.232.46.133): icmp_seq=3 ttl=51 time=37.0 ms
64 bytes from 199.232.46.133 (199.232.46.133): icmp_seq=4 ttl=51 time=37.0 ms

--- dualstack.d.sni.global.fastly.net ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 36.691/36.901/36.998/0.122 ms

ddas · November 22, 2024, 4:12am

Thanks for your response, appreciate it.
Here is what I have in the unit file.

$ grep -vE '^$|^#' /lib/systemd/system/jenkins.service
[Unit]
Description=Jenkins Continuous Integration Server
Requires=network.target
After=network.target
[Service]
Type=notify
NotifyAccess=main
ExecStart=/usr/bin/jenkins
Restart=on-failure
SuccessExitStatus=143
User=jenkins
Group=jenkins
Environment="JENKINS_HOME=/var/lib/jenkins"
WorkingDirectory=/var/lib/jenkins
Environment="JENKINS_WEBROOT=%C/jenkins/war"
Environment="JAVA_OPTS=-Djava.awt.headless=true"
Environment="JENKINS_PORT=8080"
[Install]
WantedBy=multi-user.target

We (my employer) are not ready to roll out U22 or U24 yet so looking for some way to debug further and find out the root cause why systemd is failing.

MarkEWaite · November 22, 2024, 6:01am

The /lib/systemd/system/jenkins.service contents match with the contents on my Ubuntu 22.04 system.

Does the override file contain anything suspicious?

/etc/systemd/system/jenkins.service.d/override.conf is the override file on my Ubuntu 22.04 system.

poddingue · November 22, 2024, 5:20pm

To check the status of your network service, please type systemctl is-active network.target.
If you’re looking for detailed error messages, go ahead and run systemctl start jenkins.service -l.
Want more verbose logs? Try journalctl -u jenkins -xe.
Make sure that your Java runtime is configured correctly and accessible.

ddas · November 26, 2024, 7:29pm

Perfect, thank you so much, it helped.

ddas · November 26, 2024, 7:31pm

I had to make multiple corrections in that
/etc/systemd/system/jenkins.service.d/override.conf file. It now looks something like this.

[Service]
User=ansible
Group=ansible
Environment="JAVA_OPTS=-Xms4g -Xmx16g -XX:+AlwaysPreTouch -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/jenkins/ -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+ParallelRefProcEnabled -XX:+DisableExplicitGC -XX:+UnlockDiagnosticVMOptions -XX:+UnlockExperimentalVMOptions -Xlog:gc*=info,gc+heap=debug,gc+ref*=debug,gc+ergo*=trace,gc+age*=trace:file=/var/log/jenkins/gc.log:utctime,pid,level,tags:filecount=2,filesize=100M -XX:ErrorFile=/var/log/jenkins/hs_err_%p.log -XX:+LogVMOutput -XX:LogFile=/var/log/jenkins/hs_%p.log  -Dorg.jenkinsci.plugins.pipeline.modeldefinition.parser.RuntimeASTTransformer.SCRIPT_SPLITTING_TRANSFORMATION=false -Dhudson.model.DirectoryBrowserSupport.CSP= -Djava.io.tmpdir=/var/lib/jenkins/tmp"

And finally I got it to work.

ddas · November 26, 2024, 7:42pm

I made changes to the order in which I was trying to install the packages, and launch the jenkins service with a different username other than jenkins, it’s ansible in this case.

After removing override.conf, purging packages and re-installing openjdk-11, jenkins, plugins, jenkins service came up running with jenkins user.
I then created the override.conf file with necessary changes.
chown’d the /var/lib/jenkins /var/log/jenkins /var/cache/jenkins directories to provide access to ansible user.
Restarted the jenkins service, and it came back running with ansible user as intended.

poddingue · November 26, 2024, 8:10pm

Thanks a lot for the feedback, @ddas !

Topic		Replies	Views
About the Using Jenkins category Using Jenkins	2	1659	July 1, 2024
Jenkins failed to start Ask a question	4	4641	May 18, 2023
How to fix jenkins installation error on ubuntu linux Using Jenkins	5	12847	December 13, 2022
Just installed Jenkins 2.452.3 LTS for my Ubuntu 22 but encountered some issue like service failed to start Using Jenkins question	9	546	July 19, 2024
Failed to start Jenkins Continuous Integration Server [code=exited, status=217/USER] Using Jenkins question	1	314	September 30, 2024

Systemd is failing to launch jenkins

Related topics