Hi @halkeye - thanks for the info. I’m in the same boat and a bit frazzled. I’d love to get my manually upgraded (war file) Jenkins working properly (i.e. systemctl scripts) without resorting to the yum upgrade method.
I went from 2.235.3-1.1 to 2.361.1 and finally got the instance to work, all plugins are up to date, and at least 3 similar jobs appear to work as they did before the upgrade … as described here.
I’m running RHEL 7.9 Maipo.
It sounds like you are saying the issue comes down to the style of init script used? I believe this instance has used systemctl (systemd?) from its beginning, but perhaps the issue is still related to init script format. systemctl seems to show one vendor old-style rc.d script is disabled, and there is an /etc/systemd/system/jenkins.service.d/override.conf?
which only contains:
[Service]
Environment="JAVA_OPTS=-Djava.awt.headless=true -Djava.io.tmpdir=/var/lib/jenkins/tmp"
Is it possible to fix the init script (or edit the service) in my case? Do any of the links you posted previously have info about this?
Here is info from my machine:
[root@localhost:~]$ systemctl status jenkins
● jenkins.service - LSB: Jenkins Automation Server
Loaded: loaded (/etc/rc.d/init.d/jenkins; bad; vendor preset: disabled)
Drop-In: /etc/systemd/system/jenkins.service.d
└─override.conf
Active: failed (Result: timeout) since Thu 2022-09-08 21:54:26 EDT; 9h ago
Docs: man:systemd-sysv-generator(8)
Process: 10975 ExecStart=/etc/rc.d/init.d/jenkins start (code=killed, signal=TERM)
CGroup: /system.slice/jenkins.service
├─10980 runuser -s /bin/bash jenkins -c ulimit -S -c 0 >/dev/null 2>&1 ; /usr/lib/jvm/jre-11/bin/java -Djava.awt.headless=true -Djava.io.tmpdir=/var/lib/jenkins/tmp -DJENKINS_HOME=/var/lib/jenkin...
├─10981 bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /usr/lib/jvm/jre-11/bin/java -Djava.awt.headless=true -Djava.io.tmpdir=/var/lib/jenkins/tmp -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/...
└─10982 /usr/lib/jvm/jre-11/bin/java -Djava.awt.headless=true -Djava.io.tmpdir=/var/lib/jenkins/tmp -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/je...
Sep 08 21:49:26 localhost runuser[10980]: pam_unix(runuser:session): session opened for user jenkins by (uid=0)
Sep 08 21:49:33 localhost jenkins[10975]: Starting Jenkins WARNING: An illegal reflective access operation has occurred
Sep 08 21:49:33 localhost jenkins[10975]: WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/var/cache/jenkins/war/WEB-INF/lib/groovy-all-2....g.Class,int)
Sep 08 21:49:33 localhost jenkins[10975]: WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1
Sep 08 21:49:33 localhost jenkins[10975]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
Sep 08 21:49:33 localhost jenkins[10975]: WARNING: All illegal access operations will be denied in a future release
Sep 08 21:54:26 localhost systemd[1]: jenkins.service start operation timed out. Terminating.
Sep 08 21:54:26 localhost systemd[1]: Failed to start LSB: Jenkins Automation Server.
Sep 08 21:54:26 localhost systemd[1]: Unit jenkins.service entered failed state.
Sep 08 21:54:26 localhost systemd[1]: jenkins.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
systemctl show jenkins
Type=forking
Restart=no
NotifyAccess=none
RestartUSec=100ms
TimeoutStartUSec=5min
TimeoutStopUSec=5min
WatchdogUSec=0
WatchdogTimestampMonotonic=0
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
FailureAction=none
PermissionsStartOnly=no
RootDirectoryStartOnly=no
RemainAfterExit=yes
GuessMainPID=no
MainPID=0
ControlPID=0
FileDescriptorStoreMax=0
StatusErrno=0
Result=timeout
ExecMainStartTimestampMonotonic=0
ExecMainExitTimestampMonotonic=0
ExecMainPID=0
ExecMainCode=0
ExecMainStatus=0
ExecStart={ path=/etc/rc.d/init.d/jenkins ; argv[]=/etc/rc.d/init.d/jenkins start ; ignore_errors=no ; start_time=[Thu 2022-09-08 21:49:26 EDT] ; stop_time=[Thu 2022-09-08 21:54:26 EDT] ; pid=10975 ; code=killed ; status=15/TERM }
ExecReload={ path=/etc/rc.d/init.d/jenkins ; argv[]=/etc/rc.d/init.d/jenkins reload ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
ExecStop={ path=/etc/rc.d/init.d/jenkins ; argv[]=/etc/rc.d/init.d/jenkins stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
Slice=system.slice
ControlGroup=/system.slice/jenkins.service
MemoryCurrent=18446744073709551615
TasksCurrent=18446744073709551615
Delegate=no
CPUAccounting=no
CPUShares=18446744073709551615
StartupCPUShares=18446744073709551615
CPUQuotaPerSecUSec=infinity
BlockIOAccounting=no
BlockIOWeight=18446744073709551615
StartupBlockIOWeight=18446744073709551615
MemoryAccounting=no
MemoryLimit=18446744073709551615
DevicePolicy=auto
TasksAccounting=no
TasksMax=18446744073709551615
Environment=JAVA_OPTS=-Djava.awt.headless=true -Djava.io.tmpdir=/var/lib/jenkins/tmp
UMask=0022
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=4096
LimitAS=18446744073709551615
LimitNPROC=63183
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=63183
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615
OOMScoreAdjust=0
Nice=0
IOScheduling=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SecureBits=0
CapabilityBoundingSet=18446744073709551615
AmbientCapabilities=0
MountFlags=0
PrivateTmp=no
PrivateNetwork=no
PrivateDevices=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=no
IgnoreSIGPIPE=no
NoNewPrivileges=no
SystemCallErrorNumber=0
RuntimeDirectoryMode=0755
KillMode=process
KillSignal=15
SendSIGKILL=yes
SendSIGHUP=no
Id=jenkins.service
Names=jenkins.service
Requires=basic.target system.slice
Wants=network-online.target
WantedBy=graphical.target multi-user.target
Conflicts=shutdown.target
Before=graphical.target shutdown.target multi-user.target
After=sendmail.service nmon.service time-sync.target basic.target system.slice remote-fs.target nss-lookup.target network-online.target systemd-journald.socket
Documentation=man:systemd-sysv-generator(8)
Description=LSB: Jenkins Automation Server
LoadState=loaded
ActiveState=failed
SubState=failed
FragmentPath=/run/systemd/generator.late/jenkins.service
SourcePath=/etc/rc.d/init.d/jenkins
DropInPaths=/etc/systemd/system/jenkins.service.d/override.conf
UnitFileState=bad
UnitFilePreset=disabled
InactiveExitTimestamp=Thu 2022-09-08 21:49:26 EDT
InactiveExitTimestampMonotonic=8480357492895
ActiveEnterTimestamp=Thu 2022-09-08 18:25:51 EDT
ActiveEnterTimestampMonotonic=8468142540705
ActiveExitTimestamp=Thu 2022-09-08 18:38:34 EDT
ActiveExitTimestampMonotonic=8468905179965
InactiveEnterTimestamp=Thu 2022-09-08 21:54:26 EDT
InactiveEnterTimestampMonotonic=8480657642369
CanStart=yes
CanStop=yes
CanReload=yes
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=no
IgnoreOnSnapshot=no
NeedDaemonReload=no
JobTimeoutUSec=0
JobTimeoutAction=none
ConditionResult=yes
AssertResult=yes
ConditionTimestamp=Thu 2022-09-08 21:49:26 EDT
ConditionTimestampMonotonic=8480357491932
AssertTimestamp=Thu 2022-09-08 21:49:26 EDT
AssertTimestampMonotonic=8480357491932
Transient=no
CollectMode=inactive
[root@localhost:~]$ more /etc/sysconfig/jenkins
## Path: Development/Jenkins
## Description: Jenkins Automation Server
## Type: string
## Default: "/var/lib/jenkins"
## ServiceRestart: jenkins
#
# Directory where Jenkins store its configuration and working
# files (checkouts, build reports, artifacts, ...).
#
JENKINS_HOME="/var/lib/jenkins"
## Type: string
## Default: ""
## ServiceRestart: jenkins
#
# Java executable to run Jenkins
# When left empty, we'll try to find the suitable Java.
#
JENKINS_JAVA_CMD="/usr/lib/jvm/jre-11/bin/java"
## Type: string
## Default: "jenkins"
## ServiceRestart: jenkins
#
# Unix user account that runs the Jenkins daemon
# Be careful when you change this, as you need to update
# permissions of $JENKINS_HOME and /var/log/jenkins.
#
JENKINS_USER="jenkins"
## Type: string
## Default: "false"
## ServiceRestart: jenkins
#
# Whether to skip potentially long-running chown at the
# $JENKINS_HOME location. Do not enable this, "true", unless
# you know what you're doing. See JENKINS-23273.
#
#JENKINS_INSTALL_SKIP_CHOWN="false"
## Type: string
## Default: "-Djava.awt.headless=true"
## ServiceRestart: jenkins
#
# Options to pass to java when running Jenkins.
#
JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Djava.io.tmpdir=/var/lib/jenkins/tmp"
## Type: integer(0:65535)
## Default: 8080
## ServiceRestart: jenkins
#
# Port Jenkins is listening on.
# Set to -1 to disable
#
JENKINS_PORT="8080"
## Type: string
## Default: ""
## ServiceRestart: jenkins
#
# IP address Jenkins listens on for HTTP requests.
# Default is all interfaces (0.0.0.0).
#
JENKINS_LISTEN_ADDRESS=""
## Type: integer(0:65535)
## Default: ""
## ServiceRestart: jenkins
#
# HTTPS port Jenkins is listening on.
# Default is disabled.
#
JENKINS_HTTPS_PORT=""
## Type: string
## Default: ""
## ServiceRestart: jenkins
#
# Path to the keystore in JKS format (as created by the JDK 'keytool').
# Default is disabled.
#
JENKINS_HTTPS_KEYSTORE=""
## Type: string
## Default: ""
## ServiceRestart: jenkins
#
# Password to access the keystore defined in JENKINS_HTTPS_KEYSTORE.
# Default is disabled.
#
JENKINS_HTTPS_KEYSTORE_PASSWORD=""
## Type: string
## Default: ""
## ServiceRestart: jenkins
#
# IP address Jenkins listens on for HTTPS requests.
# Default is disabled.
#
JENKINS_HTTPS_LISTEN_ADDRESS=""
## Type: integer(0:65535)
## Default: ""
## ServiceRestart: jenkins
#
# HTTP2 port Jenkins is listening on.
# Default is disabled.
#
# Notice: HTTP2 support may require additional configuration, see Winstone
# documentation for more information.
#
JENKINS_HTTP2_PORT=""
## Type: string
## Default: ""
## ServiceRestart: jenkins
#
# IP address Jenkins listens on for HTTP2 requests.
# Default is disabled.
#
# Notice: HTTP2 support may require additional configuration, see Winstone
# documentation for more information.
#
JENKINS_HTTP2_LISTEN_ADDRESS=""
## Type: integer(1:9)
## Default: 5
## ServiceRestart: jenkins
#
# Debug level for logs -- the higher the value, the more verbose.
# 5 is INFO.
#
JENKINS_DEBUG_LEVEL="5"
## Type: yesno
## Default: no
## ServiceRestart: jenkins
#
# Whether to enable access logging or not.
#
JENKINS_ENABLE_ACCESS_LOG="no"
## Type: integer
## Default: 100
## ServiceRestart: jenkins
#
# Maximum number of HTTP worker threads.
#
JENKINS_HANDLER_MAX="100"
## Type: integer
## Default: 20
## ServiceRestart: jenkins
#
# Maximum number of idle HTTP worker threads.
#
JENKINS_HANDLER_IDLE="20"
## Type: string
## Default: ""
## ServiceRestart: jenkins
#
# Folder for additional jar files to add to the Jetty class loader.
# See Winstone documentation for more information.
# Default is disabled.
#
JENKINS_EXTRA_LIB_FOLDER=""
## Type: string
## Default: ""
## ServiceRestart: jenkins
#
# Pass arbitrary arguments to Jenkins.
# Full option list: java -jar jenkins.war --help
#
JENKINS_ARGS=""
[root@localhost:~]$ more /etc/rc.d/init.d/jenkins
#!/bin/sh
#
# RedHat system statup script for Jenkins
# Based on SUSE system statup script for Jenkins
# Copyright (C) 2007 Pascal Bleser
#
# This library is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or (at
# your option) any later version.
#
# This library is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with this library; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307,
# USA.
#
###############################################################################
#
# chkconfig: 35 99 01
# description: Jenkins Automation Server
#
###############################################################################
### BEGIN INIT INFO
# Provides: jenkins
# Required-Start: $local_fs $remote_fs $network $time $named
# Should-Start: $time sendmail
# Required-Stop: $local_fs $remote_fs $network $time $named
# Should-Stop: $time sendmail
# Default-Start: 3 5
# Default-Stop: 0 1 2 6
# Short-Description: Jenkins Automation Server
# Description: Jenkins Automation Server
### END INIT INFO
# Check for missing binaries (stale symlinks should not happen)
JENKINS_WAR="/usr/lib/jenkins/jenkins.war"
test -r "$JENKINS_WAR" || { echo "$JENKINS_WAR not installed";
if [ "$1" = "stop" ]; then exit 0;
else exit 5; fi; }
# Check for existence of needed config file and read it
JENKINS_CONFIG=/etc/sysconfig/jenkins
test -e "$JENKINS_CONFIG" || { echo "$JENKINS_CONFIG not existing";
if [ "$1" = "stop" ]; then exit 0;
else exit 6; fi; }
test -r "$JENKINS_CONFIG" || { echo "$JENKINS_CONFIG not readable. Perhaps you forgot 'sudo'?";
if [ "$1" = "stop" ]; then exit 0;
else exit 6; fi; }
JENKINS_PID_FILE="/var/run/jenkins.pid"
JENKINS_LOCKFILE="/var/lock/subsys/jenkins"
# Source function library.
. /etc/init.d/functions
# Read config
[ -f "$JENKINS_CONFIG" ] && . "$JENKINS_CONFIG"
# Set up environment accordingly to the configuration settings
[ -n "$JENKINS_HOME" ] || { echo "JENKINS_HOME not configured in $JENKINS_CONFIG";
if [ "$1" = "stop" ]; then exit 0;
else exit 6; fi; }
[ -d "$JENKINS_HOME" ] || { echo "JENKINS_HOME directory does not exist: $JENKINS_HOME";
if [ "$1" = "stop" ]; then exit 0;
else exit 1; fi; }
# Search usable Java as /usr/bin/java might not point to minimal version required by Jenkins.
# see http://www.nabble.com/guinea-pigs-wanted-----Hudson-RPM-for-RedHat-Linux-td25673707.html
candidates="
/etc/alternatives/java
/usr/lib/jvm/java-1.8.0/bin/java
/usr/lib/jvm/jre-1.8.0/bin/java
/usr/lib/jvm/java-1.7.0/bin/java
/usr/lib/jvm/jre-1.7.0/bin/java
/usr/lib/jvm/java-11.0/bin/java
/usr/lib/jvm/jre-11.0/bin/java
/usr/lib/jvm/java-11-openjdk-amd64
/usr/bin/java
"
for candidate in $candidates
do
[ -x "$JENKINS_JAVA_CMD" ] && break
JENKINS_JAVA_CMD="$candidate"
done
JAVA_CMD="$JENKINS_JAVA_CMD $JENKINS_JAVA_OPTIONS -DJENKINS_HOME=$JENKINS_HOME -jar $JENKINS_WAR"
PARAMS="--logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon"
[ -n "$JENKINS_PORT" ] && PARAMS="$PARAMS --httpPort=$JENKINS_PORT"
[ -n "$JENKINS_LISTEN_ADDRESS" ] && PARAMS="$PARAMS --httpListenAddress=$JENKINS_LISTEN_ADDRESS"
[ -n "$JENKINS_HTTPS_PORT" ] && PARAMS="$PARAMS --httpsPort=$JENKINS_HTTPS_PORT"
[ -n "$JENKINS_HTTPS_KEYSTORE" ] && PARAMS="$PARAMS --httpsKeyStore=$JENKINS_HTTPS_KEYSTORE"
[ -n "$JENKINS_HTTPS_KEYSTORE_PASSWORD" ] && PARAMS="$PARAMS --httpsKeyStorePassword='$JENKINS_HTTPS_KEYSTORE_PASSWORD'"
[ -n "$JENKINS_HTTPS_LISTEN_ADDRESS" ] && PARAMS="$PARAMS --httpsListenAddress=$JENKINS_HTTPS_LISTEN_ADDRESS"
[ -n "$JENKINS_HTTP2_PORT" ] && PARAMS="$PARAMS --http2Port=$JENKINS_HTTP2_PORT"
[ -n "$JENKINS_HTTP2_LISTEN_ADDRESS" ] && PARAMS="$PARAMS --http2ListenAddress=$JENKINS_HTTP2_LISTEN_ADDRESS"
[ -n "$JENKINS_DEBUG_LEVEL" ] && PARAMS="$PARAMS --debug=$JENKINS_DEBUG_LEVEL"
[ -n "$JENKINS_HANDLER_STARTUP" ] && PARAMS="$PARAMS --handlerCountStartup=$JENKINS_HANDLER_STARTUP"
[ -n "$JENKINS_HANDLER_MAX" ] && PARAMS="$PARAMS --handlerCountMax=$JENKINS_HANDLER_MAX"
[ -n "$JENKINS_HANDLER_IDLE" ] && PARAMS="$PARAMS --handlerCountMaxIdle=$JENKINS_HANDLER_IDLE"
[ -n "$JENKINS_EXTRA_LIB_FOLDER" ] && PARAMS="$PARAMS --extraLibFolder=$JENKINS_EXTRA_LIB_FOLDER"
[ -n "$JENKINS_ARGS" ] && PARAMS="$PARAMS $JENKINS_ARGS"
if [ "$JENKINS_ENABLE_ACCESS_LOG" = "yes" ]; then
PARAMS="$PARAMS --accessLoggerClassName=winstone.accesslog.SimpleAccessLogger --simpleAccessLogger.format=combined --simpleAccessLogger.file=/var/log/jenkins/access_log"
fi
RETVAL=0
case "$1" in
start)
echo -n "Starting Jenkins "
daemon --user "$JENKINS_USER" --pidfile "$JENKINS_PID_FILE" $JAVA_CMD $PARAMS > /dev/null
RETVAL=$?
if [ $RETVAL = 0 ]; then
success
echo > "$JENKINS_PID_FILE" # just in case we fail to find it
MY_SESSION_ID=`/bin/ps h -o sess -p $$`
# get PID
/bin/ps hww -u "$JENKINS_USER" -o sess,ppid,pid,cmd | \
while read sess ppid pid cmd; do
[ "$ppid" = 1 ] || continue
# this test doesn't work because Jenkins sets a new Session ID
# [ "$sess" = "$MY_SESSION_ID" ] || continue
echo "$cmd" | grep $JENKINS_WAR > /dev/null
[ $? = 0 ] || continue
# found a PID
echo $pid > "$JENKINS_PID_FILE"
done
touch $JENKINS_LOCKFILE
else
failure
fi
echo
;;
stop)
echo -n "Shutting down Jenkins "
killproc jenkins
rm -f $JENKINS_LOCKFILE
RETVAL=$?
echo
;;
try-restart|condrestart)
if test "$1" = "condrestart"; then
echo "${attn} Use try-restart ${done}(LSB)${attn} rather than condrestart ${warn}(RH)${norm}"
fi
$0 status
if test $? = 0; then
$0 restart
else
: # Not running is not a failure.
fi
;;
restart)
$0 stop
$0 start
;;
force-reload)
echo -n "Reload service Jenkins "
$0 try-restart
;;
reload)
$0 restart
;;
status)
status jenkins
RETVAL=$?
;;
probe)
## Optional: Probe for the necessity of a reload, print out the
## argument to this init script which is required for a reload.
## Note: probe is not (yet) part of LSB (as of 1.9)
test "$JENKINS_CONFIG" -nt "$JENKINS_PID_FILE" && echo reload
;;
*)
echo "Usage: $0 {start|stop|status|try-restart|restart|force-reload|reload|probe}"
exit 1
;;
esac
exit $RETVAL
