Jenkins fails on FIPS mode enabled RHEL

Using Jenkins
1

Server: RHEL 8.6
Jenkins version: 2.332.3 (the only option in the Jenkins RHEL repo)
Java version: openjdk 11.0.15 2022-04-19 LTS

After a recent java update, the install of Jenkins where I work has broken. First, I noticed that Jenkins builds seem to have lost communication with the build node. When a build is kicked off it showed as being in progress in the build log but under “Build executor status” it shows “0 of 3 executors busy”. In addition, after kicking off a build on a particular job, that job would then show “Oops! A problem has occurred while processing the request.” when clicking into it.

To make a long story short, I tried lots of troubleshooting and then just decided that we would be better off starting from scratch since too many people had access to this installation over the past few years (before I worked at this company) and things were a mess. So, on a fresh RHEL 8.6 server (totally different server than the one above), I followed the instructions here (Linux) to install Jenkins. Again, this was a completely fresh server with nothing installed. After following the instructions, Jenkins was still broken.

The service starts fine but there are lots of Java errors in the “journalctl -xe” output. They all start with “java.lang.AssertionError: InstanceIdentity is missing its singleton”. There are also many java errors in the java system log that you access from the Jenkins web interface. Also, many of the “Manage Jenkins” buttons result in an “Oops! A problem occurred while processing the request.” error message that includes a logging ID that links you to more java errors.

I tried testing this on Azure VMs to rule out Jenkins as the problem, but all of their RHEL 8.x VMs have broken yum tools at the moment. They say they are aware of this and are working on a fix.

Has anyone tried to install Jenkins on RHEL 8.x lately? We are dead in the water without Jenkins and I do not know what else to try. I tried every Java version I could find, including installing Oracle’s java from a .rpm, but there are always java errors (Jenkins failed to even start with Oracle’s java). Any input would be greatly appreciated!

2

While it is certainly possible that Jenkins LTS installations are broken on Red Hat Enterprise Linux 8.6, that would be very surprising. I’m running Jenkins weekly on my Red Hat Enterprise Linux 8.6 without any issue. I’m confident that others are doing the same as well.

Package installation of weekly is checked regularly on the packaging job. That test automation includes:

  • almalinux-8
  • amazonlinux-2
  • centos-stream-8
  • centos-stream-9
  • debian-10
  • debian-11
  • fedora-34
  • fedora-35
  • opensuse-leap-15
  • oraclelinux-8
  • rockylinux-8
  • ubuntu-18.04
  • ubuntu-20.04
  • ubuntu-21.04
  • ubuntu-21.10
  • ubuntu-22.04

In addition to that test, there are other checks that the confirm the RPM installer is working:

Jenkins 2.332.3 is not the only option in the Jenkins rpm repository. Specific versions can be installed using additional arguments to the yum or dnf command.

1 Like
3

Also jenkins really only supports java 8 or 11 (though there’s support for newer ones). how are you confirming your java version?

1 Like
4

Do you know what command I would use to check other Jenkins versions available? I am not having much luck googling. “sudo yum list available | grep jenkins” is giving me nothing, same with a few variations I tried on that command.

5

Yeah I did read that in the docs, but I was desperate and running out of ideas so I tried every version I could find. I confirmed java version with the command “java --version”, although on the fresh server there was no java installed at the start so I know that Java 11 got installed when I ran the “sudo yum install java-11-openjdk” command as instructed on the Jenkins install page for RHEL.

6

I don’t think you’re going to solve the problem you are having by changing the Jenkins version that you are installing. Other users are running Jenkins 2.332.3 successfully on Red Hat Enterprise Linux 8.6. The conditions that are causing the failure on your system are likely specific to your environment.

As a test, I ran the following commands to confirm that I can install Jenkins 2.332.3 on Red Hat Enterprise Linux 8.6

# wget -O /etc/yum.repos.d/jenkins.repo \
    https://pkg.jenkins.io/redhat-stable/jenkins.repo
# rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
# yum upgrade
# yum install jenkins

You can list older versions of the package with the command:

# dnf --showduplicates list jenkins

You can install an older version of the package with the command:

# dnf install jenkins-2.332.2-1.1
7

I suspect that is the best observation to guide you to find the issue with your system configuration. If the Jenkins controller web page is available, you can open the “Manage Jenkins” → “System Information” page to confirm that you are running Java 8 or Java 11.

The cases in the Jenkins Jira tracker that mention InstanceIdentity is missing its singleton all seem to refer to running a very old Jenkins version with Java 9.

8

First, thank you for your help, this has been a stressful few days and I was out of ideas.

Second, here is the System Information for my installation…

System Properties

[Name ↓] [Value]
awt.toolkit sun.awt.X11.XToolkit
executable-war /usr/share/java/jenkins.war
file.encoding UTF-8
file.separator /
java.awt.graphicsenv sun.awt.X11GraphicsEnvironment
java.awt.headless true
java.awt.printerjob sun.print.PSPrinterJob
java.class.path /usr/share/java/jenkins.war
java.class.version 55.0
java.home /usr/lib/jvm/java-11-openjdk-11.0.15.0.10-2.el8_6.x86_64
java.io.tmpdir /tmp
java.library.path /usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
java.runtime.name OpenJDK Runtime Environment
java.runtime.version 11.0.15+10-LTS
java.specification.name Java Platform API Specification
java.specification.vendor Oracle Corporation
java.specification.version 11
java.vendor Red Hat, Inc.
java.vendor.url https:// www. redhat. com/
java.vendor.url.bug https:// bugzilla. redhat. com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%208&component=java-11-openjdk
java.vendor.version 18.9
java.version 11.0.15
java.version.date 2022-04-19
java.vm.compressedOopsMode Zero based
java.vm.info mixed mode, sharing
java.vm.name OpenJDK 64-Bit Server VM
java.vm.specification.name Java Virtual Machine Specification
java.vm.specification.vendor Oracle Corporation
java.vm.specification.version 11
java.vm.vendor Red Hat, Inc.
java.vm.version 11.0.15+10-LTS
javax.net.ssl.keyStore NONE
javax.net.ssl.trustStoreType pkcs12
jdk.debug release
jetty.git.hash 526006ecfa3af7f1a27ef3a288e2bef7ea9dd7e8
jna.loaded true
jna.platform.library.path /usr/lib64:/lib64:/usr/lib:/lib
jnidispatch.path /var/lib/jenkins/.cache/JNA/temp/jna18187688161931666668.tmp
line.separator
mail.smtp.sendpartial true
mail.smtps.sendpartial true
os.arch amd64
os.name Linux
os.version 4.18.0-372.9.1.el8.x86_64
path.separator :
sun.arch.data.model 64
sun.boot.library.path /usr/lib/jvm/java-11-openjdk-11.0.15.0.10-2.el8_6.x86_64/lib
sun.cpu.endian little
sun.cpu.isalist
sun.font.fontmanager sun.awt.X11FontManager
sun.io.unicode.encoding UnicodeLittle
sun.java.command /usr/share/java/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080
sun.java.launcher SUN_STANDARD
sun.jnu.encoding UTF-8
sun.management.compiler HotSpot 64-Bit Tiered Compilers
sun.os.patch.level unknown
user.country US
user.dir /var/lib/jenkins
user.home /var/lib/jenkins
user.language en
user.name jenkins
user.timezone America/New_York

Environment Variables

[Name ↓] [Value]
HOME /var/lib/jenkins
INVOCATION_ID a090d9fc518148eba5a72fb565f95aa7
JENKINS_HOME /var/lib/jenkins
JOURNAL_STREAM 9:1287379
LANG en_US.UTF-8
LOGNAME jenkins
NOTIFY_SOCKET /run/systemd/notify
PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
PWD /var/lib/jenkins
SHLVL 0
USER jenkins

This is definitely the latest version of both Java and Jenkins since I pulled them both straight from their respective repos upon installation with “yum install”.

9

I wonder if you inadvertently placed something in your /var/lib/jenkins directory that is harming the Jenkins installation. For example:

  • Are the installed plugins up to date and running current versions?
  • Are you running RHEL with an exotic setting like FIPS mode enabled or SELinux enabled?
  • Are there messages in the startup log that hint you are running out of memory?
11

I have been running “sudo rm -rf /var/lib/jenkins/” before every Jenkins installation just to be sure that no leftovers are causing issues. I also have been selecting “install recommended plugins” on first access to the new Jenkins installs.

FIPS mode is enabled on this server, it was enabled after the OS was installed. After checking, SELinux is also enabled (“getenforce” command returns “Enforcing”). Will these cause issues with Jenkins?

No messages related to memory in the startup log. Also, “free -h” tells me that only about 10% of available memory is actually in use on this server.

12

I’ve never run Jenkins with FIPS mode enabled or with SELinux enabled. There are several bug reports on issues.jenkins.io related to SELinux and FIPS:

1 Like
13

This might be [JENKINS-68662] Instance identity plugin can't encode/decode PEM in a FIPS configured OS / host - Jenkins Jira as pointed out by @MarkEWaite

The plugin tries to write RSA private key to a file which is not allowed in FIPS mode. You probably cannot use this plugin when running under FIPS (if Jenkins runs at all).

1 Like
14

Jenkins and its many plugins are not FIPS compliant out of the box[1] and trying to run it on a RedHat FIPS enabled host with the JVM in RedHat FIPS mode will leave your JVM still not in FIPS compliant mode as we use non JDK encryption routines. To save yourself from hurt I would disable FIPS on this host as it does nothing for the FIPS compliance of Jenkins and actively causes harm as you discovered.

[1] there are manual steps you can take for Jenkins and some plugins, but you will really need to check all the plugins you use, or have support from a vendor that would do this for you.

2 Likes
15

I don’t know what planet you folks live in, but on this planet there are organizations running classified programs for the government and 1) disabling is not an option and 2) Cloudbees telling cleared contractors to disable FIPS to run their tools is an instant non-starter.

1 Like
16

As far as I can tell, if disabling FIPS is not an option, then you’re unlikely to be successful running Jenkins on that host. Jenkins is not tested with Red Hat Enterprise Linux with FIPS enabled. There are documented issues with running Jenkins on Red Hat Enterprise Linux with FIPS enabled. See the earlier list in this thread for the items that I found with a preliminary search of the Jenkins issue tracker.

You’re mistakenly assuming that comments in this community forum are advice from CloudBees. They are not. Comments in this community forum are focused on Jenkins, not on products like CloudBees CI that are based on Jenkins.

If you’d like to discuss FIPS compliance with CloudBees, please discuss it with CloudBees in a commercial setting.

If you’d like to contribute FIPS compliance testing or FIPS compliance bug fixes to Jenkins, I’m sure there are people that would be happy to review your pull requests.

17

This is not just Jenkins, but pretty much all the Cloudbees CD/CI/RO products. If they don’t want to sell to government or defense, fine, but they’re shooting off a large chunk of their feet.

18

We appreciate your enthusiasm for the topic. CloudBees is actively working on a FIPS (140-2) compliant version of CloudBees CI for modern cloud platforms that can be utilized by Federal agencies as well as service providers or contractors who need to meet these requirements. This is a top priority for CloudBees. As we are working on this, some changes will need to be made in the Jenkins LTS, but these changes alone will not make Jenkins OSS FIPS compliant. Please reach out to CloudBees, or me directly at jleon@cloudbees.com, if you have any further questions or comments on the topic.

20

We don’t support vendor specific versions of jenkins (ex cloudbees) on the OSS forums/mailing lists full stop.

Its already been very clearly stated that fips mode is not supported by the OSS versions. Closing this.