Continue Discussion 29 replies
March 2023

MarkEWaite Jenkins Governance Board

Installing 2.387.1 before March 30, 2023

The new PGP key is not valid with Jenkins 2.387.1. Use the previous PGP key to install Jenkins 2.387.1 and earlier.

Installing 2.387.1 March 30, 2023 or later

The new PGP key used to sign the Jenkins LTS 2.387.1 deb and rpm files will expire March 30, 2023. Users installing Jenkins LTS 2.387.1 after March 31, 2023 may see a warning or an error noting that the PGP key has expired.

Jenkins LTS 2.387.2 (April 5, 2023) will resolve that warning, so long as the new PGP public key has been installed by following the instructions in the Linux installation page.

1 reply
March 2023 ▶ MarkEWaite

basil Jenkins Governance Board

The instructions in the blog post worked for me but printed some deprecation warnings:

$ wget -qO - https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo apt-key add -
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
OK
$

To avoid using deprecated functionality I downloaded the https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key file and put it in /usr/share/keyrings/jenkins-keyring.asc and updated /etc/apt/sources.list.d/jenkins.list with:

deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://pkg.jenkins.io/debian binary/

I have no idea if this is the best way of doing things or not, but it solved the deprecation warning I was getting on Ubuntu 22.04.2 LTS x86_64.

1 reply
March 2023

MarkEWaite Jenkins Governance Board

Thanks for detecting that mistake and reporting it. I’ve submitted a fix to the blog post to use the same key installation instructions as are used in the Linux install guides.

The install guide instructions to install the GPG public key on Debian and Ubuntu are:

$ curl -fsSL https://pkg.jenkins.io/debian/jenkins.io-2023.key | sudo tee \
  /usr/share/keyrings/jenkins-keyring.asc > /dev/null
$ echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
  https://pkg.jenkins.io/debian binary/ | sudo tee \
  /etc/apt/sources.list.d/jenkins.list > /dev/null

The instructions for key installation on Red Hat / CentOS / Alma / Rocky are:

$ sudo rpm --import https://pkg.jenkins.io/redhat/jenkins.io-2023.key
1 reply
March 2023 ▶ MarkEWaite

MarkEWaite Jenkins Governance Board

Debian workaround for LTS

The Debian installation instructions for Jenkins LTS create the following line in /etc/apt/sources.list.d/jenkins.list:

deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://pkg.jenkins.io/debian-stable binary/

That configuration assures that the jenkins-keyring is used to validate the packages from the Jenkins Debian stable repository without using that keyring for packages from any other repository. When that configuration is used with the previous PGP key on Debian 11, the error that is reported is:

Reading package lists... Done
W: GPG error: https://pkg.jenkins.io/debian-stable binary/ Release: The following signatures were invalid: EXPKEYSIG FCEF32E745F2C3D5 Jenkins Project <jenkinsci-board@googlegroups.com>
E: The repository 'https://pkg.jenkins.io/debian-stable binary/ Release' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

That results in an error because the https://pkg.jenkins.io/debian-stable repository is not signed.

The error can be temporarily converted to a warning by using the following change in /etc/apt/sources.list.d/jenkins.list:

deb [allow-insecure=yes] https://pkg.jenkins.io/debian-stable binary/

That skips the PGP signature check only for packages from the debian-stable repository. Once Jenkins 2.387.2 is released, the original configuration can be restored to use the jenkins-keyring.

The messages from apt-get are then warnings instead of errors and look like this:

Reading package lists... Done
W: GPG error: https://pkg.jenkins.io/debian-stable binary/ Release: The following signatures were invalid: EXPKEYSIG FCEF32E745F2C3D5 Jenkins Project <jenkinsci-board@googlegroups.com>
W: The repository 'https://pkg.jenkins.io/debian-stable binary/ Release' is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.

When the install is run with apt-get install jenkins, then the output will look like this:

$ sudo apt-get install jenkins
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  net-tools
The following NEW packages will be installed:
  jenkins net-tools
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 96.1 MB/96.3 MB of archives.
After this operation, 99.4 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
WARNING: The following packages cannot be authenticated!
  jenkins
Install these packages without verification? [y/N] y
Get:1 https://pkg.jenkins.io/debian-stable binary/ jenkins 2.387.1 [96.1 MB]
Fetched 96.1 MB in 5s (20.8 MB/s)
Selecting previously unselected package net-tools.
(Reading database ... 202724 files and directories currently installed.)
Preparing to unpack .../net-tools_1.60+git20181103.0eebece-1_amd64.deb ...
Unpacking net-tools (1.60+git20181103.0eebece-1) ...
Selecting previously unselected package jenkins.
Preparing to unpack .../jenkins_2.387.1_all.deb ...
Unpacking jenkins (2.387.1) ...
Setting up net-tools (1.60+git20181103.0eebece-1) ...
Setting up jenkins (2.387.1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/jenkins.service → /lib/systemd/system/jenkins.service.
1 reply
March 2023

jkie-dr

Hello, how can we be alerted of these types of changes so that we can update proactively in the future?

1 reply
March 2023 ▶ MarkEWaite

afunix

Any idea when debian-stable repo will be signed? Are we talking days, weeks, months? Do I need to put “allow-insecure=yes” workaround into ansible or wait few days for the change?

It’s been 4 days since the announcement, keys expired and still some repos are not ready… That’s far from ideal…

1 reply
March 2023

MarkEWaite Jenkins Governance Board

April 5, 2023 as stated in the blog post:

Beginning March 28, 2023, the Jenkins weekly releases will use new repository signing keys for the Linux installation packages. The same change will be made in Jenkins LTS releases beginning April 5, 2023.

March 2023

MarkEWaite Jenkins Governance Board

Some of the places that announced the change include:

The change will also be announced in the Jenkins 2.387.2 changelog and upgrade guide and in the “What’s New in Jenkins 2.387.2” live stream.

I agree that even those channels were probably not enough. This change would have been well suited to appear as a Jenkins admin monitor 3-6 months prior to the change. That would have alerted administrators that the change was coming and given them time to plan for the change.

I’ve seen a suggestion that the Debian packages be signed with multiple keys so that the transition could be easier for administrators. I think that is worth exploring as well. I’m sure there are other ideas that are worth considering as well. Keep the ideas and suggestions coming.

1 reply
March 2023 ▶ MarkEWaite

jkie-dr

Thank you for the advice! I’ll check out the RSS and other links above. Much appreciated.

April 2023

yanksyoon

Hello! Has the stable repository (LTS) been signed yet?
Even after following the instructions i’m getting

W: GPG error: https://pkg.jenkins.io/debian-stable binary/ Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY FCEF32E745F2C3D5
E: The repository 'https://pkg.jenkins.io/debian-stable binary/ Release' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
2 replies
April 2023 ▶ yanksyoon

mtughan

Hi @yanksyoon. That appears to be the signature of the old key. Note that the path to the signature file has changed and it’s now available at https://pkg.jenkins.io/debian/jenkins.io-2023.key. Testing this morning installing 2.387.2 with that signature file appears to run successfully on my Ubuntu 18.04 machine.

April 2023

MarkEWaite Jenkins Governance Board

Yes, the stable repository has been signed. The 2.387.2 release had not been run at the time you asked the question. It has run now. The instructions on the page are now updated. Thanks for checking!

We’ll host a retrospective on the challenges associated with the rotation of the PGP repository signing keys and the challenges associated with the rotation of the code signing certificate used for the MSI installer and the war file. I’m sure that improvements will be identified in that retrospective.

April 2023

chrisj

Our pre-2.387 installs are now failing. This is the first time we’ve come across this issue. I should add a new install has worked once, but we have this error on 2 other attempts (same code).

wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat/jenkins.repo
rpm --import https://pkg.jenkins.io/redhat/jenkins.io.key
yum install -y jenkins-2.359-1.1

Public key for jenkins-2.359-1.1.noarch.rpm is not installed
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: GPG check FAILED
1 reply
April 2023

MarkEWaite Jenkins Governance Board

I’m not able to duplicate that behavior on my RHEL 8 system. I removed Jenkins with sudo yum erase jenkins and then installed Jenkins 2.332.3 with sudo yum install jenkins-2.332.3-1.1. The package installs as expected, though the yum list --installed output shows an @ sign that I believe indicates the package signing key has expired or is not valid.

My machine may be different than yours because I’ve imported the new signing key with sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io-2023.key

April 2023

brandongallagher

Hi Mark, I have been trying to upgrade Jenkins and have followed your instructions in this blog, but am getting the errors below. I’m running on Ubuntu/Debian.

bgallagher:/ >sudo apt-get update
Hit:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates InRelease
Ign:3 https://pkg.jenkins.io/debian-stable binary/ InRelease
Hit:4 https://deb.nodesource.com/node_15.x xenial InRelease
Ign:5 https://pkg.jenkins.io/debian-stable binary/ Release
Hit:6 http://security.ubuntu.com/ubuntu xenial-security InRelease
Ign:7 https://pkg.jenkins.io/debian-stable binary/ Packages.diff/Index
Ign:8 https://pkg.jenkins.io/debian-stable binary/ Translation-en_US
Hit:9 http://ppa.launchpad.net/brightbox/ruby-ng/ubuntu xenial InRelease
Hit:10 https://packagecloud.io/modeanalytics/main/ubuntu xenial InRelease
Ign:11 https://pkg.jenkins.io/debian-stable binary/ Translation-en
Ign:12 https://pkg.jenkins.io/debian-stable binary/ Packages
Ign:8 https://pkg.jenkins.io/debian-stable binary/ Translation-en_US
Ign:11 https://pkg.jenkins.io/debian-stable binary/ Translation-en
Ign:12 https://pkg.jenkins.io/debian-stable binary/ Packages
Ign:8 https://pkg.jenkins.io/debian-stable binary/ Translation-en_US
Ign:11 https://pkg.jenkins.io/debian-stable binary/ Translation-en
Ign:12 https://pkg.jenkins.io/debian-stable binary/ Packages
Ign:8 https://pkg.jenkins.io/debian-stable binary/ Translation-en_US
Ign:11 https://pkg.jenkins.io/debian-stable binary/ Translation-en
Ign:12 https://pkg.jenkins.io/debian-stable binary/ Packages
Ign:8 https://pkg.jenkins.io/debian-stable binary/ Translation-en_US
Ign:11 https://pkg.jenkins.io/debian-stable binary/ Translation-en
Ign:12 https://pkg.jenkins.io/debian-stable binary/ Packages
Ign:8 https://pkg.jenkins.io/debian-stable binary/ Translation-en_US
Ign:11 https://pkg.jenkins.io/debian-stable binary/ Translation-en
Err:12 https://pkg.jenkins.io/debian-stable binary/ Packages
  server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
Reading package lists... Done
W: The repository 'https://pkg.jenkins.io/debian-stable binary/ Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: Failed to fetch https://pkg.jenkins.io/debian-stable/binary/Packages  server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
E: Some index files failed to download. They have been ignored, or old ones used instead.

I ran the steps in the blog post to get the key, but I’m not sure if that’s the problem or not. Any ideas?

1 reply
April 2023

halkeye Leader

Sounds like you have something wrong with your systems CA certificates.

Other than that maybe your system is old enough not to have letsencrypt trusted? That’s not a recent change though

May 2023

jsluck

Right now, when I look, the new key and the previous key appear to be the same key.
Has something changed on the redhat-stable repo? Older versions are no longer able to install.

1 reply
May 2023

MarkEWaite Jenkins Governance Board

I can’t see that. When I download https://pkg.jenkins.io/redhat-stable/jenkins.io-2023.key it is different from https://pkg.jenkins.io/redhat-stable/jenkins.io.key

You’ll need to provide more information about what you’re reading that causes you to see the new key and the previous key to be the same key.

1 reply
May 2023

chanda-is

Getting the same " The repository ‘Debian Jenkins Packages binary/ Release’ is not signed." on a fresh install of Ubuntu 20.04 after running the commands specified in this post:

~$  curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo tee   /usr/share/keyrings/jenkins-keyring.asc > /dev/null
~$ echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc]   https://pkg.jenkins.io/debian-stable binary/ | sudo tee   /etc/apt/sources.list.d/jenkins.list > /dev/null
~$ sudo apt update
Ign:1 https://pkg.jenkins.io/debian-stable binary/ InRelease
Hit:2 https://pkg.jenkins.io/debian-stable binary/ Release
Get:3 https://pkg.jenkins.io/debian-stable binary/ Release.gpg [833 B]
Ign:3 https://pkg.jenkins.io/debian-stable binary/ Release.gpg
 ...
Reading package lists... Done
W: GPG error: https://pkg.jenkins.io/debian-stable binary/ Release: Unknown error executing apt-key
E: The repository 'https://pkg.jenkins.io/debian-stable binary/ Release' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

This worked for me on a different system about 2 weeks ago so not sure if something changed

2 replies
May 2023

halkeye Leader

Googling says its an issue with your system, something with gpg. Which would probably make sense for a fresh install not having gpg installed by default?

1 reply
May 2023 ▶ halkeye

chanda-is

saw that, but no, it’s installed

:~$ gpg --version
gpg (GnuPG) 2.2.19
libgcrypt 1.8.5
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

May 2023

MarkEWaite Jenkins Governance Board

I just created a fresh Ubuntu 20.04 machine on Google Cloud, installed Java 17, and followed the installation instructions. Installation proceeded without an issue.

One possible difference is that Google Cloud installation images are probably installed with the most recent Ubuntu 20.04 packages. Have you updated your operating system to assure that it has the most recent Ubuntu 20.04 updates installed?

2 replies
May 2023 ▶ MarkEWaite

chanda-is

Well apt upgrade shows no packages are available to update, but I’m not running Java 17 (wouldn’t think that would make a difference). But since it’s working for you, it leads me to think that IT did something to this system before turning it over to me so I’ll dig some more. Thanks for the help.

May 2023 ▶ MarkEWaite

jsluck

This looks to have resolved itself overnight. I don’t really understand what went on, but it might have been a CDN issue or something as I could pull the correct cert from one device and on another it would pull the 2023 cert rather than the older one. Most odd. Glitch in a Matrix?

May 2023 ▶ MarkEWaite

rsareth

Did you try with CIS Ubuntu Linux 20.04 LTS Benchmark v1.1.0.24 - Level 1 on AWS ? For me, it doesn’t work.

1 reply
May 2023

halkeye Leader

At this point I’m tempted to lock the thread. It’s not up to mark to spend time and money testing every flavor of Linux, especially ones that are 3+ years old.

That being said, recommend you create a jira ticket with any steps you’ve run, anything you can note about your network. The output of commands you ran.

“It doesn’t work” is never enough to get help

1 reply
May 2023 ▶ rsareth

rsareth

After reading the man-page of apt-key, on CIS Ubuntu Linux 20.04 LTS, I had to do these:

$ sudo apt-key add /usr/share/keyrings/jenkins-keyring.asc
OK
$ apt-key list
/etc/apt/trusted.gpg
--------------------
[...]

pub   rsa4096 2023-03-27 [SC] [expires: 2026-03-26]
      6366 7EE7 4BBA 1F0A 08A6  9872 5BA3 1D57 EF59 75CA
uid           [ unknown] Jenkins Project <jenkinsci-board@googlegroups.com>
sub   rsa4096 2023-03-27 [E] [expires: 2026-03-26]

[...]
deb https://pkg.jenkins.io/debian-stable binary/
$ sudo apt update
Hit:1 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:3 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu focal-backports InRelease
Ign:4 https://pkg.jenkins.io/debian-stable binary/ InRelease
Hit:5 https://pkg.jenkins.io/debian-stable binary/ Release
Hit:6 http://security.ubuntu.com/ubuntu focal-security InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
May 2023 ▶ halkeye

rsareth

You are right. My bad.

I tested a new approach for the CIS flavor in my previous post and it seems to work. I will need to test further in my script to be sure.

May 2023

chanda-is

just wanted to post in case it helps someone else - I was able to resolve my issue by doing

sudo chmod 644 /usr/share/keyrings/jenkins-keyring.asc

something I found from another post on a different package having the same issue