Beginning March 28, 2023, the Jenkins weekly releases will use new repository signing keys for the Linux installation packages. The same change will be made in Jenkins LTS releases beginning April 5, 2023.
Instructions will be included in the changelog for the release and in a blog post. Conversations about the change can be posted as replies to this topic.
The new PGP key is not valid with Jenkins 2.387.1. Use the previous PGP key to install Jenkins 2.387.1 and earlier.
The new PGP key used to sign the Jenkins LTS 2.387.1 deb and rpm files will expire March 30, 2023. Users installing Jenkins LTS 2.387.1 after March 31, 2023 may see a warning or an error noting that the PGP key has expired.
Jenkins LTS 2.387.2 (April 5, 2023) will resolve that warning, so long as the new PGP public key has been installed by following the instructions in the Linux installation page.
The instructions in the blog post worked for me but printed some deprecation warnings:
$ wget -qO - https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo apt-key add -
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
OK
$
To avoid using deprecated functionality I downloaded the https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key file and put it in /usr/share/keyrings/jenkins-keyring.asc and updated /etc/apt/sources.list.d/jenkins.list with:
deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://pkg.jenkins.io/debian binary/
I have no idea if this is the best way of doing things or not, but it solved the deprecation warning I was getting on Ubuntu 22.04.2 LTS x86_64.
Thanks for detecting that mistake and reporting it. Iāve submitted a fix to the blog post to use the same key installation instructions as are used in the Linux install guides.
The install guide instructions to install the GPG public key on Debian and Ubuntu are:
$ curl -fsSL https://pkg.jenkins.io/debian/jenkins.io-2023.key | sudo tee \
/usr/share/keyrings/jenkins-keyring.asc > /dev/null
$ echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
https://pkg.jenkins.io/debian binary/ | sudo tee \
/etc/apt/sources.list.d/jenkins.list > /dev/null
The instructions for key installation on Red Hat / CentOS / Alma / Rocky are:
$ sudo rpm --import https://pkg.jenkins.io/redhat/jenkins.io-2023.key
The Debian installation instructions for Jenkins LTS create the following line in /etc/apt/sources.list.d/jenkins.list:
deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://pkg.jenkins.io/debian-stable binary/
That configuration assures that the jenkins-keyring is used to validate the packages from the Jenkins Debian stable repository without using that keyring for packages from any other repository. When that configuration is used with the previous PGP key on Debian 11, the error that is reported is:
Reading package lists... Done
W: GPG error: https://pkg.jenkins.io/debian-stable binary/ Release: The following signatures were invalid: EXPKEYSIG FCEF32E745F2C3D5 Jenkins Project <jenkinsci-board@googlegroups.com>
E: The repository 'https://pkg.jenkins.io/debian-stable binary/ Release' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
That results in an error because the https://pkg.jenkins.io/debian-stable repository is not signed.
The error can be temporarily converted to a warning by using the following change in /etc/apt/sources.list.d/jenkins.list:
deb [allow-insecure=yes] https://pkg.jenkins.io/debian-stable binary/
That skips the PGP signature check only for packages from the debian-stable repository. Once Jenkins 2.387.2 is released, the original configuration can be restored to use the jenkins-keyring.
The messages from apt-get are then warnings instead of errors and look like this:
Reading package lists... Done
W: GPG error: https://pkg.jenkins.io/debian-stable binary/ Release: The following signatures were invalid: EXPKEYSIG FCEF32E745F2C3D5 Jenkins Project <jenkinsci-board@googlegroups.com>
W: The repository 'https://pkg.jenkins.io/debian-stable binary/ Release' is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
When the install is run with apt-get install jenkins, then the output will look like this:
$ sudo apt-get install jenkins
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
net-tools
The following NEW packages will be installed:
jenkins net-tools
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 96.1 MB/96.3 MB of archives.
After this operation, 99.4 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
WARNING: The following packages cannot be authenticated!
jenkins
Install these packages without verification? [y/N] y
Get:1 https://pkg.jenkins.io/debian-stable binary/ jenkins 2.387.1 [96.1 MB]
Fetched 96.1 MB in 5s (20.8 MB/s)
Selecting previously unselected package net-tools.
(Reading database ... 202724 files and directories currently installed.)
Preparing to unpack .../net-tools_1.60+git20181103.0eebece-1_amd64.deb ...
Unpacking net-tools (1.60+git20181103.0eebece-1) ...
Selecting previously unselected package jenkins.
Preparing to unpack .../jenkins_2.387.1_all.deb ...
Unpacking jenkins (2.387.1) ...
Setting up net-tools (1.60+git20181103.0eebece-1) ...
Setting up jenkins (2.387.1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/jenkins.service ā /lib/systemd/system/jenkins.service.
Hello, how can we be alerted of these types of changes so that we can update proactively in the future?
Any idea when debian-stable repo will be signed? Are we talking days, weeks, months? Do I need to put āallow-insecure=yesā workaround into ansible or wait few days for the change?
Itās been 4 days since the announcement, keys expired and still some repos are not readyā¦ Thatās far from idealā¦
April 5, 2023 as stated in the blog post:
Beginning March 28, 2023, the Jenkins weekly releases will use new repository signing keys for the Linux installation packages. The same change will be made in Jenkins LTS releases beginning April 5, 2023.
Some of the places that announced the change include:
The change will also be announced in the Jenkins 2.387.2 changelog and upgrade guide and in the āWhatās New in Jenkins 2.387.2ā live stream.
I agree that even those channels were probably not enough. This change would have been well suited to appear as a Jenkins admin monitor 3-6 months prior to the change. That would have alerted administrators that the change was coming and given them time to plan for the change.
Iāve seen a suggestion that the Debian packages be signed with multiple keys so that the transition could be easier for administrators. I think that is worth exploring as well. Iām sure there are other ideas that are worth considering as well. Keep the ideas and suggestions coming.
Thank you for the advice! Iāll check out the RSS and other links above. Much appreciated.
Hello! Has the stable repository (LTS) been signed yet?
Even after following the instructions iām getting
W: GPG error: https://pkg.jenkins.io/debian-stable binary/ Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY FCEF32E745F2C3D5
E: The repository 'https://pkg.jenkins.io/debian-stable binary/ Release' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
Hi @yanksyoon. That appears to be the signature of the old key. Note that the path to the signature file has changed and itās now available at https://pkg.jenkins.io/debian/jenkins.io-2023.key. Testing this morning installing 2.387.2 with that signature file appears to run successfully on my Ubuntu 18.04 machine.
Yes, the stable repository has been signed. The 2.387.2 release had not been run at the time you asked the question. It has run now. The instructions on the page are now updated. Thanks for checking!
Weāll host a retrospective on the challenges associated with the rotation of the PGP repository signing keys and the challenges associated with the rotation of the code signing certificate used for the MSI installer and the war file. Iām sure that improvements will be identified in that retrospective.
Our pre-2.387 installs are now failing. This is the first time weāve come across this issue. I should add a new install has worked once, but we have this error on 2 other attempts (same code).
wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat/jenkins.repo
rpm --import https://pkg.jenkins.io/redhat/jenkins.io.key
yum install -y jenkins-2.359-1.1
Public key for jenkins-2.359-1.1.noarch.rpm is not installed
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: GPG check FAILED
Iām not able to duplicate that behavior on my RHEL 8 system. I removed Jenkins with sudo yum erase jenkins and then installed Jenkins 2.332.3 with sudo yum install jenkins-2.332.3-1.1. The package installs as expected, though the yum list --installed output shows an @ sign that I believe indicates the package signing key has expired or is not valid.
My machine may be different than yours because Iāve imported the new signing key with sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io-2023.key
Hi Mark, I have been trying to upgrade Jenkins and have followed your instructions in this blog, but am getting the errors below. Iām running on Ubuntu/Debian.
bgallagher:/ >sudo apt-get update
Hit:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates InRelease
Ign:3 https://pkg.jenkins.io/debian-stable binary/ InRelease
Hit:4 https://deb.nodesource.com/node_15.x xenial InRelease
Ign:5 https://pkg.jenkins.io/debian-stable binary/ Release
Hit:6 http://security.ubuntu.com/ubuntu xenial-security InRelease
Ign:7 https://pkg.jenkins.io/debian-stable binary/ Packages.diff/Index
Ign:8 https://pkg.jenkins.io/debian-stable binary/ Translation-en_US
Hit:9 http://ppa.launchpad.net/brightbox/ruby-ng/ubuntu xenial InRelease
Hit:10 https://packagecloud.io/modeanalytics/main/ubuntu xenial InRelease
Ign:11 https://pkg.jenkins.io/debian-stable binary/ Translation-en
Ign:12 https://pkg.jenkins.io/debian-stable binary/ Packages
Ign:8 https://pkg.jenkins.io/debian-stable binary/ Translation-en_US
Ign:11 https://pkg.jenkins.io/debian-stable binary/ Translation-en
Ign:12 https://pkg.jenkins.io/debian-stable binary/ Packages
Ign:8 https://pkg.jenkins.io/debian-stable binary/ Translation-en_US
Ign:11 https://pkg.jenkins.io/debian-stable binary/ Translation-en
Ign:12 https://pkg.jenkins.io/debian-stable binary/ Packages
Ign:8 https://pkg.jenkins.io/debian-stable binary/ Translation-en_US
Ign:11 https://pkg.jenkins.io/debian-stable binary/ Translation-en
Ign:12 https://pkg.jenkins.io/debian-stable binary/ Packages
Ign:8 https://pkg.jenkins.io/debian-stable binary/ Translation-en_US
Ign:11 https://pkg.jenkins.io/debian-stable binary/ Translation-en
Ign:12 https://pkg.jenkins.io/debian-stable binary/ Packages
Ign:8 https://pkg.jenkins.io/debian-stable binary/ Translation-en_US
Ign:11 https://pkg.jenkins.io/debian-stable binary/ Translation-en
Err:12 https://pkg.jenkins.io/debian-stable binary/ Packages
server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
Reading package lists... Done
W: The repository 'https://pkg.jenkins.io/debian-stable binary/ Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: Failed to fetch https://pkg.jenkins.io/debian-stable/binary/Packages server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
E: Some index files failed to download. They have been ignored, or old ones used instead.
I ran the steps in the blog post to get the key, but Iām not sure if thatās the problem or not. Any ideas?
Sounds like you have something wrong with your systems CA certificates.
Other than that maybe your system is old enough not to have letsencrypt trusted? Thatās not a recent change though
Right now, when I look, the new key and the previous key appear to be the same key.
Has something changed on the redhat-stable repo? Older versions are no longer able to install.
I canāt see that. When I download https://pkg.jenkins.io/redhat-stable/jenkins.io-2023.key it is different from https://pkg.jenkins.io/redhat-stable/jenkins.io.key
Youāll need to provide more information about what youāre reading that causes you to see the new key and the previous key to be the same key.