Infrastructure Team Meeting - January 13, 2026

[dduportal (Damien Duportal)]

2026 01 13 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @MarkEWaite (Mark Waite)
• @hervelemeur (Hervé Le Meur)

Announcements

1. Jenkins Weekly Releases
   • Last Week: 2.545 - You're invited to talk on Matrix
     • With the Docker image packaging introducing Windows Server 2022 image replacing the 2019 one
     • We weren’t aware it also featured removal of JDK17 (e.g. JDK21 is required). Side effects on packages and Docker images, but fixed on Wednesday 07 January - see Jenkins 2.545 AlmaLinux package forces Java 17 dependency, breaks startup requiring Java 21 · Issue #729 · jenkinsci/packaging · GitHub
   • This Week: 2.546 - You're invited to talk on Matrix
     • Fixes on packaging
     • No more JDK17 published
2. Announcements:
   • Team capacity:
     • Jay is off including tomorrow
     • Damien off for ~6 weeks on the first February, maybe earlier
   • Infra Roadmap: Jenkins Roadmap
     • New priorities:
       • Add support for Windows 2025 and deprecated 2019
       • JDK25 as runtime for our controllers
       • Get rid of Puppet (in favor of Ansible)
       • new: ci.jenkins.io => cloud billing and stability

Upcoming Calendar

• Next Weekly: 2026-01-20 - 2.547
• Next LTS: 2026-01-21 - 2.541.1
  • Helpdesk opened to collect all related changes: LTS 2.541.1 release · Issue #4945 · jenkins-infra/helpdesk · GitHub
  • Proposition to publish Docker controller LTS images from a stable-<release-line> branch like in packaging and release: GitHub · Where software is built
  • Need Jenkins infra team review on the incoming changelog and upgrade guide (ping both Herve and Damien)
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • 2026-01-24: repo.jenkins-ci.org TLS certificate expires
    • [repo.jenkins-ci.org (Artifactory)] TLS certificates expires on `2026-01-24` · Issue #4893 · jenkins-infra/helpdesk · GitHub => Herve and Damien in pair
    • Certificate should be done later today
  • 2026-01-21: Update Center generation fails as the certificate is 30 days before expiration => Damien
    • Update-center / crawler root certificate expires on 21 March 2026 (failing job on 21 February 2026) · Issue #4952 · jenkins-infra/helpdesk · GitHub => Damien (needs access to the CA authority)
    • Note: we can temporarily resume Update Center by decreasing the 30 days threshold in trusted.ci.jenkins.io freestyle job if we are stuck
  • 2026-01-27: VPN CRL expires
    • [private.vpn.jenkins.io] VPN CRL expires on 2026-01-27 => Herve
  • 2026-01-28: Expiration of the Digital Oceans PATs used by Terraform on infra.ci.jenkins.io
    • [infra.ci.jenkins.io] Track Digital Ocean PAT expiration (2026-01-28) used by terraform with updatecli · Issue #4943 · jenkins-infra/helpdesk · GitHub => Jay with Herve in pair
• Next major event:
  • Contributor Summit + FOSDEM in Brussels, 30/31 Jan and 1st Feb
    • We have a booth for Jenkins at FOSDEM
    • Agenda for Summit is still WiP. Bring ideas (in community.jenkins.io)
    • Jenkins Contributor Summit on Jan 30, 2026 - Call for topics and ideas
    • Mark is confirming with people based on the prioritized list from the Jenkins Board

Cloud Budgets

• Azure CDF - Remaining: $24.000k on 1st January 2026 - monthly threshold set at $6.0k (good until May at the current rate)

  • October: $5.5k (invoice)
  • November: $5.7k (invoice)
  • December: $5.4k (invoice, previously forecasted at $6k)
  • January: $2.2k (forecast at $5.4k)

• DigitalOcean - Remaining $10,549.16 until January 02, 2027 (20 months left at current rate)

  • October: $788.81 (invoice)
  • November: $831.38 (invoice)
  • December: $454.60 (invoice, previously forecasted at $585)
  • January: $144.80 (forecast at ~$345)

• AWS:

  • CloudBees:

    • October: $580
    • November: $530
    • December: $384
    • January: $107 (forecast at $290)
      • A few resources to delete, snapshot of VMLs to transfer from AWS EC2 to Azure blob storage and we can close

  • Sponsored account - $54,613.14 left (31 December 2025) until 31 May 2027=> 5 months remaining at this rate

    • October: $6.9k
    • November: $5.9k
    • December: $10.7 (previously forecasted at $9.5k way higher than usual)
      • Huge increase due to EC2 plugin bug
    • January: $4.4k (forecast at $11.5k)
      • Still really high, ci.jenkins.io is once again at risk if we don’t decrease the consumption or if we don’t move it somewhere else before end of April 2026
      • Need an issue ASAP for the EC2 agents GC (Damien)
      • Mark opened an issue on the EC2 plugin: ci.jenkins.io EC2 agents sometimes hang and fail to connect with SSH key not found · Issue #1990 · jenkinsci/ec2-plugin · GitHub
      • Also: we have to request sponsoring renewal

• Jfrog Artifactory Usage

  • Storage: 1.53 Tb - (was 1.53TB last time)
  • Bandwidth:
    • October: 16.67 Tb
    • November: 15.9 Tb
    • December: 30.34Tb
      • A single IP address consumed 12Tb in December
      • JFrog didn’t block this address until this month (unless this consumer changed its IP?)
    • January: 6.27 TB Tb (forecast: 16 Tb)
      • Sounds like the abuser stopped abusing
      • One of the top 5 is an Ip from ci.jenkins.io. Not that much compared to external abusers.
        • Could it be ACP?
        • Priority on EC2 costs first

Notes

• Done:

  • Support:
    • Internal Server Error when trying to log in to Uplink
      • Looks like we have to fix our Azure based PgSQL (upgrade)
        • Issue to open
    • Add NotMyFault to uplink
      • Initial requirement: statistics
    • [ci.jenkins.io] Use of s390x permanent agent
    • Maintainer permission for lemeurherve on jenkinsci/packaging
    • Add new members to gsoc team
    • [pkg.origin.jenkins.io] puppet agent keeps updating the GPG
    • Required access for jayfranco999 and lemeurherve On Azure CDF
  • Keep infrastructure sane and maintainable
    • [trusted.ci.jenkins.io] Add arm64 agent templates
  • Keep infrastructure up to date
    • [infra.ci.jenkins.io] Azure Credential to allow writing to reports.jenkins.io expired on 2025-12-23
    • Renew ci.jenkins.io API token for incremental-publisher service before 10 January 2026

• Work in Progress:

  • ci.jenkins.io is responding slowly

    • Incident today, still in progress: LLM scrapping + BOM build => failure
    • Does it relate to EC2 costs?
      • If bandwidth cost increase: no question we enable login and block
      • If EC2 cost increase, could incrrase the effects of ec2 plugin bug and generate more rebuilds
    • Time to re-prioritize enabling logging

  • LTS 2.541.1 release

    • [pkg.jenkins.io] Unify RPM Jenkins packages (RedHat/OpenSUSE)
    • [pkg.jenkins.io/release.ci.jenkins.io] Jenkins Packaging GPG key expires on the 26 March 2026
    • Add .war.asc to get.jenkins.io

  • [pkg.jenkins.io] migrate the pkg.origin.jenkins.io service from AWS VM to Azure publick8s

    • One last item to finish: archiving the old VM disk in Azure

  • Add support for Windows 2025 agents

    • Windows 2025 agents are available on all controllers except release.ci (expected)
    • All tests from Herve on ci.jio (Docker build, plugin build, etc.) are all green
    • Next steps:
      • Update our “homemade” monitorings (acceptance tests on ci.jio, “Infra-Health agent” pipelines on trusted/cert.ci)
      • Then will follow up on distinct issues (less noisy and easier to track):
        • [must] Move to Windows 2025 as default for Windows builds
        • [nice] Evaluate if we can run Docker 2022 Windows containers on 2025
        • [must] Deprecate 2019

  • Support

    • Creating new key set fails on Windows agents - Error 5: Access is denied
      • First outcome: we have to fix Windows user createion for ec2 Windows agents on ci.jio (Damien + Herve)
      • James (thanks!) helped us, we are working on a hotfix for OpenSSH + Windows + CryptoAPI
    • Hosting of CSP compatibility “microsite” repo
      • Waiting for Daniel’s feedback on Herve’s proposal to host the site in a simple way with autonomy
    • docs.jenkins.io nginx configuration leaks the nginx version
    • decomission (or rename) docs.jenkins.io?
      • Need to add a list of task to change docs.jenkins.io (Fastly + docs.origin.jenkins.io in Kubernetes) to “alpha.docs.jenkins.io” (no more Fastly, Kubernetes only)
    • Make Kris Stern an admin for all docs-related Jenkins repos
      • No more action required, waiting for Kris for closing
    • qualys-cs plugin bundles closed-source library
      • Can be removed from milestone: no more infrastructure action, only follow up task from JenSec is needed
    • Migrate core issues to GitHub
      • Closeable?
      • Can be removed from milestone: no infrastructure required action
    • Automated process for a plugin maintainer to request migration from Jira to GitHub issues
      • Can be removed from milestone: no infrastructure required action
    • Github Copilot organisation space
      • Closeable?
      • Can be removed from milestone: no infrastructure required action

  • Keep infrastructure sane and maintainable

    • [docker-openvpn] Track OpenVPN User Certificate Expiration with updatecli
      • Jay is off, delayed
      • Should be good to go for his first PR
    • No (or not visible) work done, delayed, low priority, keeping in milestone
      • [stats.jenkins.io/infra-statistics] Move “data for the usage stats site” generation (from anonymized data) out from Andrew machine
      • Spot reclamation caused build failures in core build
      • Setup a job/set of jobs to allow performing maintenance operations (cron, GCs of resources, etc.)
        • [staging.pkg.origin.jenkins.io/staging.get.jenkins.io] Garbage collect the inactive branches
        • [get.jenkins.io,mirrors.updates.jenkins.io] Resume GeoIP database weekly update
        • Ensure the BOM cache filler runs successfully and is easier to use and monitor
      • Bunch of bad links on https://updates.jenkins.io/ and https://get.jenkins.io
      • [incrementals.jenkins.io/ci.jenkins.io] Outage of incrementals due to releasebot user’s API token reset
      • github-jenkinsci-permissions-report.json hasn’t been updated since Sep 12

  • Keep infrastructure up to date

    • [repo.jenkins-ci.org (Artifactory)] TLS certificates expires on 2026-01-24
    • [private.vpn.jenkins.io] VPN CRL expires on 2026-01-27
      • New
    • [infra.ci.jenkins.io] Track Digital Ocean PAT expiration (2026-01-28) used by terraform with updatecli
      • New
    • [pkg.jenkins.io/release.jenkins.io] Certificate signing the MSI Jenkins package expires on 16 May 2026
      • No work done, delayed, low priority
    • Set up 2.539+ CSP protection on weekly.ci.jenkins.io, remove csp plugin and associated configuration
      • No work done, delayed, low priority

• Issues staying in backlog/triage:

  • Use JDK25 for our Jenkins controllers
  • Upgrade to Kubernetes 1.34
  • Inform of Ingress NGINX Retirement in March 2026
  • [Azure Deprecation] Convert your OS disks to Standard SSD or Premium SSD before 8 September 2028
  • [Azure] Merge webservices data storage accounts into a single one with NFS v4.1
  • [updates.jenkins.io] set up mirrorbits to keep serving update-center from mirrors even if outdated
  • private docker image registry for staging core security releases
  • Tombstone Puppet (and replace it by something else)
  • Chinese jenkins site incorrect site redirection
  • Add a real-world job to weekly.ci.jenkins.io
  • [ci.jenkins.io] Monitor and Garbage collect data volume of the DockerHub registry mirror and EC pull through cache
  • Move collection of stats out from Kohsuke’s home
  • Support [skip ci] on default branch
  • Create build for jenkinsci/winp on release ci server
  • [Update Center] HTTP/404 on /current/updates/*.json* links
  • Add monitoring for CD secrets updates
  • Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)
  • [INFRA-3046] Monitor Jenkins mirrors Age

• Issues added to the next milestone:

  • Update-center / crawler root certificate expires on 21 March 2026 (failing job on 21 February 2026)
  • Permission for jenkins.io hlemeur to manage cert.ci.jenkins.io
  • Maven 3.9.12 upgrade campaign
  • Add Rajiv Singh to Copy Editors and Infra Stats teams
  • ci.jenkins.io Windows build fails with unexpected errors (out of memory, etc.)
  • Artifact caching proxy fails with ‘No route to host’ on Jenkins core build