Server: RHEL 8.6
Jenkins version: 2.332.3 (the only option in the Jenkins RHEL repo)
Java version: openjdk 11.0.15 2022-04-19 LTS
After a recent java update, the install of Jenkins where I work has broken. First, I noticed that Jenkins builds seem to have lost communication with the build node. When a build is kicked off it showed as being in progress in the build log but under “Build executor status” it shows “0 of 3 executors busy”. In addition, after kicking off a build on a particular job, that job would then show “Oops! A problem has occurred while processing the request.” when clicking into it.
To make a long story short, I tried lots of troubleshooting and then just decided that we would be better off starting from scratch since too many people had access to this installation over the past few years (before I worked at this company) and things were a mess. So, on a fresh RHEL 8.6 server (totally different server than the one above), I followed the instructions here (Linux) to install Jenkins. Again, this was a completely fresh server with nothing installed. After following the instructions, Jenkins was still broken.
The service starts fine but there are lots of Java errors in the “journalctl -xe” output. They all start with “java.lang.AssertionError: InstanceIdentity is missing its singleton”. There are also many java errors in the java system log that you access from the Jenkins web interface. Also, many of the “Manage Jenkins” buttons result in an “Oops! A problem occurred while processing the request.” error message that includes a logging ID that links you to more java errors.
I tried testing this on Azure VMs to rule out Jenkins as the problem, but all of their RHEL 8.x VMs have broken yum tools at the moment. They say they are aware of this and are working on a fix.
Has anyone tried to install Jenkins on RHEL 8.x lately? We are dead in the water without Jenkins and I do not know what else to try. I tried every Java version I could find, including installing Oracle’s java from a .rpm, but there are always java errors (Jenkins failed to even start with Oracle’s java). Any input would be greatly appreciated!
While it is certainly possible that Jenkins LTS installations are broken on Red Hat Enterprise Linux 8.6, that would be very surprising. I’m running Jenkins weekly on my Red Hat Enterprise Linux 8.6 without any issue. I’m confident that others are doing the same as well.
Package installation of weekly is checked regularly on the packaging job. That test automation includes:
In addition to that test, there are other checks that the confirm the RPM installer is working:
Do you know what command I would use to check other Jenkins versions available? I am not having much luck googling. “sudo yum list available | grep jenkins” is giving me nothing, same with a few variations I tried on that command.
Yeah I did read that in the docs, but I was desperate and running out of ideas so I tried every version I could find. I confirmed java version with the command “java --version”, although on the fresh server there was no java installed at the start so I know that Java 11 got installed when I ran the “sudo yum install java-11-openjdk” command as instructed on the Jenkins install page for RHEL.
I don’t think you’re going to solve the problem you are having by changing the Jenkins version that you are installing. Other users are running Jenkins 2.332.3 successfully on Red Hat Enterprise Linux 8.6. The conditions that are causing the failure on your system are likely specific to your environment.
As a test, I ran the following commands to confirm that I can install Jenkins 2.332.3 on Red Hat Enterprise Linux 8.6
I suspect that is the best observation to guide you to find the issue with your system configuration. If the Jenkins controller web page is available, you can open the “Manage Jenkins” → “System Information” page to confirm that you are running Java 8 or Java 11.
The cases in the Jenkins Jira tracker that mention InstanceIdentity is missing its singleton all seem to refer to running a very old Jenkins version with Java 9.
I have been running “sudo rm -rf /var/lib/jenkins/” before every Jenkins installation just to be sure that no leftovers are causing issues. I also have been selecting “install recommended plugins” on first access to the new Jenkins installs.
FIPS mode is enabled on this server, it was enabled after the OS was installed. After checking, SELinux is also enabled (“getenforce” command returns “Enforcing”). Will these cause issues with Jenkins?
No messages related to memory in the startup log. Also, “free -h” tells me that only about 10% of available memory is actually in use on this server.
Jenkins and its many plugins are not FIPS compliant out of the box and trying to run it on a RedHat FIPS enabled host with the JVM in RedHat FIPS mode will leave your JVM still not in FIPS compliant mode as we use non JDK encryption routines. To save yourself from hurt I would disable FIPS on this host as it does nothing for the FIPS compliance of Jenkins and actively causes harm as you discovered.
 there are manual steps you can take for Jenkins and some plugins, but you will really need to check all the plugins you use, or have support from a vendor that would do this for you.