Certificate is NOT trusted when installing Jenkins

rburchdish · January 14, 2022, 11:26pm

Hi all,

We had an issue with our instance of Jenkins on our Ubuntu server which purged our configs and packages.

Because of that, we are attempting to re-install Jenkins on this server. However, we continue to get an error when running sudo apt-get update

Err:6 https://pkg.jenkins.io/debian-stable binary/ Release
  Certificate verification failed: The certificate is NOT trusted. The certificate chain uses expired certificate.  Could not handshake: Error in the certificate verification. [IP: 151.101.194.133 443]

We have followed all the steps on the Jenkins Ubuntu install page:

We have also made sure to remove Jenkins from the apt list and add back all the necessary files (once again by following the steps on the Jenkins page).

The steps we have tried:

curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo tee \ /usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \ https://pkg.jenkins.io/debian-stable binary/ | sudo tee \ /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt-get update ← This is the issue where we get the error mentioned above.

I have also tried:
wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | apt-key add -

We are running java version 8 on Ubuntu 20.04

halkeye · January 14, 2022, 11:29pm

Sounds like your system is old enough to not handle letsencypt certs.

MarkEWaite · January 14, 2022, 11:30pm

Usually when there is a report that the certificate is not trusted, it is because the operating system list of certificates is out of date. Be sure that you’ve run apt-get update && apt-get upgrade on that operating system so that it downloads and installs the latest certificate authorities.

rburchdish · January 14, 2022, 11:39pm

@MarkEWaite @halkeye thanks for the reply.

We have run both update and upgrade but are still getting the cert issue.

We have also made sure the ca-certificate was updated by running sudo apt install ca-certiticates

halkeye · January 14, 2022, 11:59pm

I can confirm its valid and working. You may want to confirm a certificate package was updated for your OS version. (You need to update your local certs before running the update-ca-certificates function)

openssl s_client -connect pkg.jenkins.io:443 2>/dev/null | openssl x509 -noout -dates
notBefore=Dec 13 11:47:49 2021 GMT
notAfter=Mar 13 11:47:48 2022 GMT

 curl -v https://pkg.jenkins.io/ >/dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 151.101.194.133:443...
* TCP_NODELAY set
* Connected to pkg.jenkins.io (151.101.194.133) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [106 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [4020 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=pkg.jenkins.io
*  start date: Dec 13 11:47:49 2021 GMT
*  expire date: Mar 13 11:47:48 2022 GMT
*  subjectAltName: host "pkg.jenkins.io" matched cert's "pkg.jenkins.io"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x564685811c90)
} [5 bytes data]
> GET / HTTP/2
> Host: pkg.jenkins.io
> user-agent: curl/7.68.0
> accept: */*
>
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
} [5 bytes data]
< HTTP/2 200
< server: Apache/2.4.29 (Ubuntu)
< content-type: text/html;charset=UTF-8
< accept-ranges: bytes
< date: Fri, 14 Jan 2022 23:58:06 GMT
< via: 1.1 varnish
< age: 3
< x-served-by: cache-sea4444-SEA
< x-cache: HIT
< x-cache-hits: 1
< x-timer: S1642204687.724884,VS0,VE0
< vary: Accept-Encoding
< strict-transport-security: max-age=300
< content-length: 3329
<
{ [1164 bytes data]
100  3329  100  3329    0     0  79261      0 --:--:-- --:--:-- --:--:-- 79261
* Connection #0 to host pkg.jenkins.io left intact

rburchdish · January 15, 2022, 12:17am

@halkeye

I recreated the ca-cert
sudo update-ca-certificates -f

It looks like I received the same output when attempting to connect to jenkins.io

curl -v https://pkg.jenkins.io/ >/dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 151.101.2.133:443...
* TCP_NODELAY set
* Connected to pkg.jenkins.io (151.101.2.133) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [106 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [4020 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=pkg.jenkins.io
*  start date: Dec 13 11:47:49 2021 GMT
*  expire date: Mar 13 11:47:48 2022 GMT
*  subjectAltName: host "pkg.jenkins.io" matched cert's "pkg.jenkins.io"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x564a14e8edb0)
} [5 bytes data]
> GET / HTTP/2
> Host: pkg.jenkins.io
> user-agent: curl/7.68.0
> accept: */*
> 
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
} [5 bytes data]
< HTTP/2 200 
< server: Apache/2.4.29 (Ubuntu)
< content-type: text/html;charset=UTF-8
< accept-ranges: bytes
< date: Sat, 15 Jan 2022 00:14:51 GMT
< via: 1.1 varnish
< age: 506
< x-served-by: cache-mdw17348-MDW
< x-cache: HIT
< x-cache-hits: 1
< x-timer: S1642205692.906762,VS0,VE1
< vary: Accept-Encoding
< strict-transport-security: max-age=300
< content-length: 3329
< 
{ [1161 bytes data]
100  3329  100  3329    0     0  26212      0 --:--:-- --:--:-- --:--:-- 26212
* Connection #0 to host pkg.jenkins.io left intact

Excuse me for not knowing, but how would I update my local certs?

halkeye · January 15, 2022, 2:45am

I don’t know offhand. update-ca-certificates takes the certs in the directory and installs them properly. One of the debian packages, maybe ca-certificates is the one that installs the latest ones for debian.

I would recommend googling for your version of linux (debian, ubuntu, arch, etc) and the version, and how to update it to support the letsencrypt root certificates.

rburchdish · January 17, 2022, 3:20pm

@halkeye Alright I will try to get that figured out.

Thanks for the help, we were able to get another instance of Jenkins running on a separate VM.

drullar · May 20, 2022, 6:26pm

Hey,

I have the same issue and have came to this post numerous times just because there wasn’t any other post out there.

As @MarkEWaite previously posted the DST Root CA X3 Expiration (September 2021) - Let's Encrypt Let’s Encrypt CA has expired. So a possible solution that worked for me, would be to check if there is a line in your /etc/ca-certificates.conf file and if there is comment it out by adding an exclamation mark.
And then run “sudo update-ca-certificates” - that will update the changes that you just made. After that try running sudo apt update once again.

Found the solution here: https://groups.google.com/g/jenkinsci-users/c/uvts55BHsE8?pli=1

rashtecq · March 4, 2023, 9:19pm

I am facing this issue on Ubuntu(22.04) which is hosted on Azure.
I followed all instructions mentioned/available on internet but still seeing the same error message like below.

azureuser@azu-svl-xxxx$ sudo apt-get install jenkins -y
Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
The following packages were automatically installed and are no longer required:
libllvm13 libvulkan1 libwayland-client0 libxcb-randr0 mesa-vulkan-drivers
Use ‘sudo apt autoremove’ to remove them.
The following NEW packages will be installed:
jenkins
0 upgraded, 1 newly installed, 0 to remove and 13 not upgraded.
Need to get 93.7 MB of archives.
After this operation, 94.3 MB of additional disk space will be used.

Err:1 Debian Jenkins Packages binary/ jenkins 2.375.3
Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown. Could not handshake: Error in the certificate verification. [IP: 146.112.56.46 443]
E: Failed to fetch https://ftp.halifax.rwth-aachen.de/jenkins/debian-stable/jenkins_2.375.3_all.deb Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown. Could not handshake: Error in the certificate verification. [IP: 146.112.56.46 443]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Tried so far:

I have reinstalled ca-certificate package multiple time
executed apt update and upgrade
Copied ca-certificate.crt from one of working instance of Jenkins but still same issue

I would really appreciate any help as I have spent almost a week on this but solution till now

MarkEWaite · March 4, 2023, 9:53pm

I used the following command to confirm that the SSL certificate on ftp.halifax.rwth-aachen.de is valid:

echo | \
  openssl s_client -showcerts -servername  ftp.halifax.rwth-aachen.de -connect ftp.halifax.rwth-aachen.de:443 2>/dev/null | \
  openssl x509 -inform pem -noout -text

Maybe you can try the same command and see if you get the same results?

I posted my results to a gist at

gist.github.com

https://gist.github.com/MarkEWaite/4317c7e5a3bcf908171d5e0d548a4351

ssl-output-for-ftp.halifax.rwth-aachen.de

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            27:53:a9:e8:f6:66:2d:e8:79:e3:70:46
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = DE, O = Verein zur Foerderung eines Deutschen Forschungsnetzes e. V., OU = DFN-PKI, CN = DFN-Verein Global Issuing CA
        Validity
            Not Before: Sep 28 11:13:31 2022 GMT
            Not After : Oct 29 11:13:31 2023 GMT

This file has been truncated. show original

That command is taken from

I’ve also confirmed from my web browser that a valid certificate is used on https://ftp.halifax.rwth-aachen.de

halkeye · March 5, 2023, 6:09am

Try confirming the date is right. SSL handshake needs an accurate clock.

Also check if you have any sort of proxy set or not set

rashtecq · March 5, 2023, 9:53am

Hi Mark,
Thanks for you suggestion. I have executed the same commond as you and compared the output. In my output issuer is Issuer: O = Cisco, CN = Cisco Umbrella Secondary SubCA lon-SG
is that correct one ? and also when I check in browser it shows certificate is not valid. Please see below snapshot.

rashtecq · March 5, 2023, 10:22am

output:

MarkEWaite · March 5, 2023, 2:30pm

It seems like something is intercepting your requests and altering them. You might try a google search for that specific issuer. When I searched, it found this article that might point you in the right direction.

rashtecq · March 6, 2023, 2:09pm

Thanks Marks for direction. I have solved this issue now.
I have installed cisco-umbrella-root-certificate following below link

and

Topic		Replies	Views
Failed to fetch https://pkg.jenkins.io/debian-stable/binary/jenkins_2.332.3_all.deb Ask a question	5	2425	May 4, 2023
I am unable to insatall jenkins on ubuntu 20.04 instance Community question	1	1094	July 28, 2023
Package Update Issue Community	2	235	October 24, 2023
Lets encrypt server certificate verification failed Using Jenkins	9	6090	December 30, 2021
Unable to update to 2.387.2 Ask a question question	3	1624	July 7, 2023

Related Topics