Windows Based Jenkins just started having certificate problems while attempting to run updates

Hello Mark,

Wouldn’t it make more sense for you guys to tell us what destination to use? Our Firewalls are not blocking any traffic to https://updates.jenkins.io, we are getting redirected to a different mirror from your side

image

We did test powershell commands to directly reach out to the repository and it work fine, so is a matter on how windows based Jenkins tries to download the files

I’m sorry, but that doesn’t explain why you are the only user reporting this issue. There are many, many installations of Jenkins all over the world (more than 250000) and yet you’re the only one reporting an issue with accessing the update center. I don’t see the issue from my location in the U.S. Other users in the U.S. don’t see the issue from their location. Other users don’t see the issue from other locations outside the U.S.

It seems to me that there is something different about your site and your networking compared to other users. I think that you’ll need to work with your networking people to understand why requests are not working from your location.

For what it is worth, we have an API endpoint listing all IPs and domains for all mirrors for both the Update Center (updates.jenkins.io which serves the index and issues some redirects) and get.jenkins.io: https://reports.jenkins.io/infrastructure/latest/index.json.

The content of this resource is kept up to date when changing.

Your screenshot shows a bluecoat configuration item. This kind of network appliance are usually doing what I call a “TLS-man-in-the-middle”: it transparently terminates TLS and re-issue a new TLS handshake with the remote server.

Since we cannot reproduce your issue (I tried on Wind. Server 2022 and 2025, from both my ISP in Eurorpe, Azure and AWS US East datacenter), it is 100% sure your bluecoat appliance which seems to react from request coming from your Jenkins JVM process but not from powershell (usually the HTTP requests headers are used by bluecoat security appliances to “triage” outgoing requests).

As we saw together, the problem is not on the updates.jenkins.io domain, but on the get.jenkins.io domain. The updates.jenkins.io is properly performed on your case as per your logs posted on your initial message.
The issue is with the get.jenkins.io connection from your Jenkins JVM, 99% sure it’s the bluecoat (It’s, alas, a “classic” in corporate environments). I guess the appliance is missing the usual certificate chain used on get.jenkins.io OR on one of the mirrors to which you are redirected.

Like Mark said, we suggest you either see with your network admin (or people in charge of the bluecoat machine) or run Jenkins in a non-constrained network.

Hello @MarkEWaite our Jenkins environment is specific to a Windows 2022 server, we understand that you have a big footprint over the world, but we are very specific and there was at least one more person reporting this problem that was sent to this thread.

@dduportal i will re-engage my network team, we did test that my servers can in fact reach out to the repos and are able to manually download plugins, but i will ask again.

As Damien said, all mirror IP addresses are kept up to date at https://reports.jenkins.io/infrastructure/latest/index.json, worth checking by your network team that they’re accessible on your side.

Hello @MarkEWaite our Jenkins environment is specific to a Windows 2022 server, we understand that you have a big footprint over the world, but we are very specific and there was at least one more person reporting this problem that was sent to this thread.

Please check Unable to download and install plugins - #3 by dduportal (The other thread I initially linked): it has a totally different error message. The can’t even reach updates.jenkins.io URL so they don’t get a redirect (yes devil is in the details… of logs :wink: ) while you issue here is with get.jenkins.io’s TLS negociation where “something” in the middle between your server and the remote server does remove some TLS intermediate certificate chain “in some cases” (e.g. when having some specific HTTP request headers).

we did test that my servers can in fact reach out to the repos and are able to manually download plugins, but i will ask again.

I have no doubt you did. But everything points at your network setup:

  • Only your network has the issue (I can’t reproduce at all with fresh Windows 2022 VMs)
  • There is a Bluecoat appliance in your network which is notoriously known to perform TLS man in the middle => and your issue is with TLS certificate chains.

My educated guess (been in your place years ago) is that your powershell has access to the “Bluecoat TLS certificate chain” in Windows while your JVM does not. Hence you need your bluecoat admin to works with you tu unblock you as their setup is not really RFC standard (and we can’t really help on the Jenkins infra side)

I got a reply from my network team: “I have validated the logs for the URLs updates.jenkins.io and mirrors.updates.jenkins.io, which are categorized as “Technology/Internet” by BlueCoat. These URLs are not being decrypted at the proxy level based on the current configuration. They are also allowed through the firewall and show a healthy volume of traffic.
Since the traffic is passing transparently and no TLS decryption is in place, the proxy should not interfere with certificate validation.
However, I observed an issue with get.jenkins.io: the certificate revocation status shows a validation error. Specifically, the CRL endpoint http://r12.c.lencr.org/33.crl returns an HTTP 500 error.
For this error, you will need to contact the certificate vendor. I am attaching the certificate error details for your reference.

That is a location provided by Let’s Encrypt. Refer to their documentation:

Hello, my network team followed up again, this time they mentioned errors with the other sites, please review the screenshots below and provide feedback:

Hello @jpinto thanks for coming back at us. We won’t do anything about these revocation messages as it is only an improvement on our TLS position. Nothing changed in this area in the past 13 months so the issue is clearly on your side and highly specific to this environment.

TLS traffic should not be decoded between the Jenkins JVM and our servers: not only it is a security issue (as it breaks the end to end connection and means the content can be modified during transfer) but also it breaks the foundations of HTTP + TCP protocols.
Your network admins must allow traffic from<->to all get.jenkins.io and updates.jenkins.io hostnames and IPs without restriction otherwise there isn’t much solution to solve your problem.

If your network believe there is a security issue on the Jenkins servers, they are welcome to report the security problem following Reporting Security Vulnerabilities so the Jenkins Security team (and eventually the Jenkins Infrastructure team) may fix (or may not if not considered a security problem).

In addition: http://r12.c.lencr.org/33.crl does answer HTTP/200 on my side (my europen ISP, but also from AWS US East 1, Azure US East 2, DigitalOcean in Germany and India).

What data would you require from us to provide to you as my network team states that we are not decrypting/decoding the traffic? The problem only started around the time your team introduced TLS and we haven’t change anything on our end, everything was working fine and our servers were properly updating. We can capture a PCAP for your review, or what else would you need?

You have to fix your JVM process and./or the netowrk so that it uses the Windows keystore: the powershell commands are working to the specified URLs so it means an standard HTTP request from this server works.
I suggest you start by removing all “custom” keystore options passed to the Jenkins JVM.

Then with pcap traces you can compare the outgoing requests between Powershell and Jenkins: are there different headers/HTTP protocol negociation which differ? Do you see different behaviors in the bluecoat between both cases?

=> that is the differential analysis you have to perform. We can’t help you more as your environment most probably have specific behavior or setup as we cannot reproduce this problem outside your environment.

The error stack trace on jenkins hints at missing intermediate certificate chains: start with this. The JVM is expected to rely on your operating system to retrieve certificate chains for LetsEncrypt certificates since years out of the box. Powershell on this machine is able to, so you should be able to search and find the right setting.

Maybe using a simple helloworld.java which emits a request and see if this simple case reproduce the error on the same JVM?

Hello Jenkins Team, after several troubleshooting sessions and a long security process we did find a specific rule in our Bluecoats that was in fact decrypting the traffic to the update sites.
We now have the proper rules and security approvals to be able to download the updates needed, this thread helped us provide the necessary feedback for them.
Thank you All for your assistance.

3 Likes