No space left on device is reported on restart

imsolanki · July 19, 2023, 9:28am

Hi all! I hope everyone is doing great,

i came up with some similar issue, I almost done all the available things on the internet even the solution provided here that is yum upgrade as I am using centOS 7.6 but nothing worked for me the error is still same.

[root@localhost log]# systemctl status jenkins
● jenkins.service - LSB: Jenkins Automation Server
   Loaded: loaded (/etc/rc.d/init.d/jenkins; bad; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2023-07-17 23:34:43 IST; 2s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 28719 ExecStart=/etc/rc.d/init.d/jenkins start (code=exited, status=1/FAILURE)
    Tasks: 72
   Memory: 1.8G
   CGroup: /system.slice/jenkins.service
           ├─26370 /bin/sh -xe /tmp/jenkins3788405557627827271.sh
           ├─26373 python3 main_exec_file.py
           ├─26762 /bin/python /bin/ansible-playbook ./play_books/build_image.yml
           ├─26876 /bin/python /bin/ansible-playbook ./play_books/build_image.yml
           ├─26890 /bin/sh -c /usr/bin/python /root/.ansible/tmp/ansible-tmp-1689616762.75-75003044585972/AnsiballZ_command.py && sleep 0
           ├─26891 /usr/bin/python /root/.ansible/tmp/ansible-tmp-1689616762.75-75003044585972/AnsiballZ_command.py
           ├─28722 runuser -s /bin/bash root -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/var/lib/jenkins -jar /u...
           ├─28723 bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkin...
           └─28724 /etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins....

Jul 17 23:34:43 localhost.localdomain jenkins[28719]: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Jul 17 23:34:43 localhost.localdomain jenkins[28719]: at java.lang.reflect.Method.invoke(Method.java:498)
Jul 17 23:34:43 localhost.localdomain jenkins[28719]: at Main._main(Main.java:342)
Jul 17 23:34:43 localhost.localdomain jenkins[28719]: at Main.main(Main.java:117)
Jul 17 23:34:43 localhost.localdomain runuser[28746]: pam_unix(runuser:session): session closed for user root
Jul 17 23:34:43 localhost.localdomain jenkins[28719]: [FAILED]
Jul 17 23:34:43 localhost.localdomain systemd[1]: jenkins.service: control process exited, code=exited status=1
Jul 17 23:34:43 localhost.localdomain systemd[1]: Failed to start LSB: Jenkins Automation Server.
Jul 17 23:34:43 localhost.localdomain systemd[1]: Unit jenkins.service entered failed state.
Jul 17 23:34:43 localhost.localdomain systemd[1]: jenkins.service failed.

On the other hand my jenkins UI is working fine and pipeline is running successfully , the only thing it is doing wrong is the date which is of the day it screwed up.

when I am trying to restart the service it is throwing the error as

[root@localhost log]# systemctl restart jenkins
Error: No space left on device
Job for jenkins.service failed because the control process exited with error code. See "systemctl status jenkins.service" and "journalctl -xe" for details.

the available space is

[root@localhost log]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root  118G   98G   20G  84% /
devtmpfs                 3.8G     0  3.8G   0% /dev
tmpfs                    3.8G  144K  3.8G   1% /dev/shm
tmpfs                    3.8G   98M  3.7G   3% /run
tmpfs                    3.8G     0  3.8G   0% /sys/fs/cgroup
/dev/sda1               1014M  180M  835M  18% /boot
/dev/loop0               4.8M  4.8M     0 100% /var/lib/snapd/snap/yq/1787
/dev/loop1               182M  182M     0 100% /var/lib/snapd/snap/postman/179
/dev/loop3                56M   56M     0 100% /var/lib/snapd/snap/core18/2284
/dev/loop4               4.8M  4.8M     0 100% /var/lib/snapd/snap/yq/1792
/dev/loop2               182M  182M     0 100% /var/lib/snapd/snap/postman/178
/dev/loop6                56M   56M     0 100% /var/lib/snapd/snap/core18/2409
/dev/loop5                47M   47M     0 100% /var/lib/snapd/snap/snapd/16010
/dev/loop7                47M   47M     0 100% /var/lib/snapd/snap/snapd/16292
/dev/mapper/centos-home   19G   18G  809M  96% /home
tmpfs                    766M  4.0K  766M   1% /run/user/42
tmpfs                    766M   52K  766M   1% /run/user/0

above is the image which is showing that jenkins pipeline is in running state only.

when I am trying to access the jenkins.log it is having the below info.

Running from: /usr/lib/jenkins/jenkins.war
Exception in thread "Jenkins initialization thread" java.lang.NoClassDefFoundError: hudson/util/HudsonFailedToLoad
	at hudson.WebAppMain$3.run(WebAppMain.java:261)
Caused by: java.lang.ClassNotFoundException: hudson.util.HudsonFailedToLoad
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:538)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 1 more
java.io.IOException: Failed to start Jetty
	at winstone.Launcher.<init>(Launcher.java:206)
	at winstone.Launcher.main(Launcher.java:405)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at Main._main(Main.java:342)
	at Main.main(Main.java:117)
Caused by: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8089
	at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
	at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
	at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
	at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.server.Server.doStart(Server.java:401)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at winstone.Launcher.<init>(Launcher.java:202)
	... 7 more
Caused by: java.net.BindException: Address already in use
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:461)
	at sun.nio.ch.Net.bind(Net.java:453)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85)
	at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
	... 14 more
Running from: /usr/lib/jenkins/jenkins.war
Exception in thread "Jenkins initialization thread" java.lang.NoClassDefFoundError: hudson/util/HudsonFailedToLoad
	at hudson.WebAppMain$3.run(WebAppMain.java:261)
Caused by: java.lang.ClassNotFoundException: hudson.util.HudsonFailedToLoad
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:538)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 1 more
java.io.IOException: Failed to start Jetty
	at winstone.Launcher.<init>(Launcher.java:206)
	at winstone.Launcher.main(Launcher.java:405)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at Main._main(Main.java:342)
	at Main.main(Main.java:117)
Caused by: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8089
	at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
	at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
	at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
	at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.server.Server.doStart(Server.java:401)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at winstone.Launcher.<init>(Launcher.java:202)

help would be really appreciated.

MarkEWaite · July 22, 2023, 9:15am

There are several problems on your system that are visible from the log file you posted. I’ve made my comments based on the information you’ve posted.

Not using systemd

Jenkins 2.332.1 switched Jenkins from using System V init to use systemd with its Linux package installers for Debian, Ubuntu, Red Hat, Alma, openSUSE, Rocky, and more. The LTS Upgrade Guide describes that transition and how to adapt your environment to the transition.

The output of your system indicates that your system is still running with System V init instead of using systemd. That likely means you have been upgrading Jenkins by replacing the war file instead of using the upgrade commands of the operating system package manager.

There is a blog post about it as well at

There is also a video introduction for RPM based distributions like Red Hat Enterprise Linux, Alma Linux, Rocky Linux, Oracle Linux, and Amazon Linux.

There is also a video introduction for deb based distributions like Debian and Ubuntu

Red Hat Enterprise Linux 7 and derivatives end of life Nov 2023

CentOS 7.9 is the most recent release of the CentOS 7 series. Even that release is very old.

Red Hat Enterprise Linux 7 and its derivatives like CentOS 7, Scientific Linux 7, and Oracle Linux 7 will no longer be supported by the Jenkins project after Nov 2023. The blog post describes more details:

File systems full or nearly full

The file systems that are listed at 100% capacity are unexpected (at least for me). I’m also quite surprised to see that snapd is being used on a CentOS 7 system. The snapcraft install page says that it is supported on CentOS 7.6 and later, but I have no experience with snap on CentOS 7.

The /home volume at 96% of capacity looks like a potential problem. Having only 800 free MB on an 18 GB volume is likely to cause performance issues and risks that you’ll run out of space.

The / volume at 84% of capacity looks like a potential problem as well.

Jenkins started by Ansible

The process listing that you provide seems to indicate that the Jenkins process is somehow being started by an Ansible playbook rather than being started by the usual init process. I’ve not seen others use that technique, nor is it described in the Jenkins documentation.

When you choose to use a novel technique to start the Jenkins controller, you are less likely to find as much help from others.

Topic		Replies	Views
Jenkins upgrade Issue in Centos 7 INC Ask a question	5	1489	September 27, 2022
Noob - Builds Fail on Node, OSError: [Errno 28] No space left on device Ask a question question	5	1422	April 12, 2024
Start Jenkins command starts but does not return back to shell on centos Ask a question question	2	1220	April 24, 2022
Jenkins initialization thread/52 died Ask a question	5	369	July 5, 2024
Space Storage issue Ask a question question , docker	3	3564	September 23, 2022

No space left on device is reported on restart

Not using systemd

Red Hat Enterprise Linux 7 and derivatives end of life Nov 2023

File systems full or nearly full

Jenkins started by Ansible

Related topics