Infrastructure Team Meeting - December 16, 2025

[lemeurherve (Hervé Le Meur)]

2025 12 16 Jenkins Infra Meeting

Attendees

• @poddingue (Bruno Verachten)
• @hervelemeur (Hervé Le Meur)
• @MarkEWaite (Mark Waite)

Announcements

1. Jenkins Weekly Releases
   • Last Week: 2.541 part of the security advisory
   • This Week: 2.542 (started on time at 4H30am UTC, failed to a code mistake in packaging, hotfixed and restarted since, released)
2. Announcements:
   • Jenkins Digicert Code Signing Certificate expires in 4 months: renew the signer certificate for jenkins · Issue #3323 · jenkins-infra/helpdesk · GitHub
     • New delivery system requires a physical token
     • Mark started investigation to use Microsoft’s GitHub action with a signing certificate instead
     • Helpdesk to open, to be dealt before May 26th of 2026
     • Try a GitHub project in packaging repository
     • Will need a different delivery process
   • After pkg.origin.jenkins.io migration to Azure, we should renew GPG key before March 2026 so we could insert it in January LTS release
     • Previous renewal: GPG key expires on March the 30th · Issue #3457 · jenkins-infra/helpdesk · GitHub
     • infra team is autonomous to renew it, but this requires extended communication
       • Ex: Announce deb & rpm signing key change
   • Team holidays (e.g. expect slower than usual):
     • Jay is off for 3 weeks. Back on the 29 Dec. 2025.
     • Mark, Herve and Damien off from 22 Dec. 2025 afor 2 weeks, back on 5th January
   • Infra Roadmap: Jenkins Roadmap
     • Current Priority: pkg.origin.jenkins.io migration
     • Roadmap deserves an update for beginning December (Herve + Damien)

Upcoming Calendar

• Next Weekly: 2.543 (next week on 23th December or next year on 6th January?)
  • Mark willing to monitor it
  • Infra team to decide
    • Message jenkinsci-dev mailing list with decision
• Next LTS: 2.541.1-rc on 7th January 2026, 2.541.1 on 21th January 2026
  • Unified RPM
  • New GPG key (see below)
  • Is it acceptable that the first RC doesn’t include this new GPG key? (Used to sign installers)
    • If not, we have to hurry up including this new GPG key in weekly 2.543
    • Acceptable
• Next Security Release as per jenkinsci-advisories: none announced
• Upcoming credentials expirations (~3 weeks):
  • 2025-12-23:
    • [infra.ci.jenkins.io] Azure Credential for updatecli (issue to create, automatic PR: Azure AD Application password for updatecli in `infra.ci.jenkins.io` expires on `2025-12-23T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #1261 · jenkins-infra/azure · GitHub)
    • [infra.ci.jenkins.io] Azure Credential to deploy docs.jenkins.io (issue to create, automatic PR: New end date for `docs.jenkins.io` File Share service principal writer on `infra.ci.jenkins.io` (current: "2025-12-23T00:00:00Z") by jenkins-infra-updatecli[bot] · Pull Request #1259 · jenkins-infra/azure · GitHub)
    • [trusted.ci.jenkins.io] Azure Credential to deploy javadoc.jenkins.io (issue to create, automatic PR: Azure File Share Principal `javadoc.jenkins.io` on `trusted.ci.jenkins.io` expires on `2025-12-23T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #1260 · jenkins-infra/azure · GitHub)
  • 2025-12-31: [trusted.ci] Artifactory Admin Token for RPU expires (issue to create, only an artifactory admin e.g. Damien, Daniel, Darin)
    • Last renewal: [trusted.ci.jenkins.io] RPU Artifactory API token expires the `2025-10-14` · Issue #4809 · jenkins-infra/helpdesk · GitHub
  • 2026-01-02: [infra.ci.jenkins.io] Azure Credential used by Packer Image builds expires (issue to create, automatic PR is expected in 2 weeks)
    • Previous issue for rotation was: [infra.ci.jenkins.io] Azure Credential used by Packer Image builds expires the 05 November 2025 · Issue #4850 · jenkins-infra/helpdesk · GitHub
  • 2026-01-24: repo.jenkins-ci.org TLS certificate expires
    • We already have the new 1 year certificate
    • Account on myJFrog to be created for Herve so he can proceed here
    • [repo.jenkins-ci.org (Artifactory)] TLS certificates expires on `2026-01-24` · Issue #4893 · jenkins-infra/helpdesk · GitHub
• Next major event:
  • Contributor Summit + FOSDEM in Brussels, 30/31 Jan and 1st Feb
    • We have a booth for Jenkins at FOSDEM
    • Agenda for Summit is still WiP. Bring ideas (in community.jenkins.io)
    • Jenkins Contributor Summit on Jan 30, 2026 - Call for topics and ideas
    • Mark is confirming with people based on the prioritized list from the Jenkins Board

Cloud Budgets

• Azure CDF - Remaining: ~$9.0k (30 Nov.) for 2025 - monthly threshold set at $6.0k

  • September: $5.8k (invoice)
  • October: $5.5k (invoice)
  • November: $5.7k (waiting for final invoice)
  • December: $2.7k (forecast at $6.0k)

• DigitalOcean - Remaining $11,122.37 until January 02, 2026 (12 months left at current rate)

  • September: $365 (invoice)
  • October: $788.81 (invoice)
  • November: $831.38 (invoice)
  • December: $264.12 (forecast at $585)
  • Remaining: $10,884.44

• AWS:

  • CloudBees:

    • September: $709
    • October: $580
    • November: $530
    • December: $199 (forecast at $449)

  • Sponsored account - $65,282.32 left (30 November 2025) until 2027 ($5,282.32 until 31 Jan. 2027 and $60,000 until 31 May 2027)=> ~11 months remaining at this rate (Nov.)

    • September: $5.4k
    • October: $6.9k
    • November: $5.9k
    • December: $4.6 (forecast at $9.5k way higher than usual)
      • ci.jenkins ddos
      • ec2 plugin (still visible)
      • opportunity for an EC2 GC? (if ssh key is ci.jio’s or packer and + 6 hours age, then delete it)

• Jfrog Artifactory Usage

  • Storage: 1.49TB - steady
  • Bandwidth:
    • September: 19.31 Tb
    • October: 16.67 Tb
    • November: 15.9 Tb (yay!)
    • December: 12.24Tb (forecast at 24.5Tb)

Notes

• Done:

  • Support
    • Windows build of Jenkins core fails with “out of memory” on ci.jenkins.io
    • CD release fails due to Non-resolvable import POM
    • No successful builds for docker-plugin-site-issues reported to GH
    • dnf5 update fails with gpgcheck=1
  • Keep infrastructure sane and maintainable
    • [release.ci.jenkins.io] Windows packaging fails to clone jenkinsci/packaging with SSH
    • Required permissions and accesses for lemeurherve to update Terraform Credentials (Backends and Cloud APIs) in jenkins-infra/terraform-states
  • Keep infrastructure up to date
    • Update ci.jenkins.io, trusted.ci, cert.ci and release.ci to latest LTS version 2.528.3
    • [Terraform Credentials] Expirations of multiple credentials (backends, API) in jenkins-infra/terraform-states
    • Upgrade to Kubernetes 1.33

• Closed as not planned:

  • Request for Permission to become a maintainer.

• Work in Progress:

  • [pkg.jenkins.io] migrate the pkg.origin.jenkins.io service from AWS VM to Azure publick8s

    • [release.ci.jenkins.io/trusted.ci.jenkins.io] Ensure Core Package build only copy package indexes/websites to pkg.origin.jenkins.io VM
      • [release.ci.jenkins.io] Packaging need to decache the pkg.jenkins.io at the end of the build (at the same time as www.jenkins.io)
        • Closable when current weekly finishes
      • https://get.jenkins.io/war-stable/latest/ does not point to latest stable release
        • Closable when current weekly finishes
    • Proposed migration date: Thursday 18th at 8h30am UTC
      • Let’s go

  • Support

    • ci.jenkins.io is responding slowly
      • No DoS since last week
      • TODO: resume work on enabling login
      • We could also include that deny list to reject AI bots: GitHub - ai-robots-txt/ai.robots.txt: A list of AI agents and robots to block.
        • Check if some of previous DoS IPs are included in that list
    • Bunch of bad links on https://updates.jenkins.io/ and https://get.jenkins.io
      • Require pkg.origin.jenkins.io migration
    • Migrate core issues to GitHub
    • Migrate core components issues to GitHub
    • [incrementals.jenkins.io/ci.jenkins.io] Outage of incrementals due to releasebot user’s API token reset
      • Delayed, need a reproduction case for Jesse Glick
    • github-jenkinsci-permissions-report.json hasn’t been updated since Sep 12
      • Delayed, waiting for Azure credits to avoid spot

  • Keep infrastructure sane and maintainable

    • [Sponsorships] Renew DigitalOcean sponsorship for 2026
      • Closable, we just received a mail confirmation today
      • TODO: confirm that new expiration date is effective
    • Required access for jayfranco999 and lemeurherve On Azure CDF
    • Required permissions and access for lemeurherve and jayfranco999 to operate on Kubernetes upgrades
      • Some details to be completed by Damien before closing it
    • Required permissions and access for lemeurherve to update CloudFlare R2 tokens of updates.jenkins.io
      • Waiting for Hervé to renew ZIP in pair with Damien before closing it
    • Setup a job/set of jobs to allow performing maintenance operations (cron, GCs of resources, etc.)
      • [get.jenkins.io,mirrors.updates.jenkins.io] Resume GeoIP database weekly update
      • Ensure the BOM cache filler runs successfully and is easier to use and monitor
      • [staging.pkg.origin.jenkins.io/staging.get.jenkins.io] Garbage collect the inactive branches
      • Delayed, focus on pkg & Kubernetes 1.33
    • Set up 2.539+ CSP protection on weekly.ci.jenkins.io, remove csp plugin and associated configuration
      • One remaining plugin to fix before closing: embeddable-build-status
    • [stats.jenkins.io/infra-statistics] Move “data for the usage stats site” generation (from anonymized data) out from Andrew machine
      • Delayed

  • Keep infrastructure up to date

    • [repo.jenkins-ci.org (Artifactory)] TLS certificates expires on 2026-01-24
    • [trusted.ci.jenkins.io] Update Center credential for Cloudflare R2 expires on 2025-12-04
    • Add support for Windows 2025 agents

• Issues to triage (section to be deleted after triage)::

• Issues staying in backlog/triage:

  • Inform of Ingress NGINX Retirement in March 2026
  • Automated process for a plugin maintainer to request migration from Jira to GitHub issues
  • [Azure Deprecation] Convert your OS disks to Standard SSD or Premium SSD before 8 September 2028
  • [Azure] Merge webservices data storage accounts into a single one with NFS v4.1
  • [updates.jenkins.io] set up mirrorbits to keep serving update-center from mirrors even if outdated
  • private docker image registry for staging core security releases
  • [Azure]: credentials less Service Principal
  • Tracking Issue for Groovy Script Conversion in RPU
  • Tombstone Puppet (and replace it by something else)
  • Chinese jenkins site incorrect site redirection
  • Add a real-world job to weekly.ci.jenkins.io
  • [ci.jenkins.io] Monitor and Garbage collect data volume of the DockerHub registry mirror and EC pull through cache
  • Move collection of stats out from Kohsuke’s home
  • Support [skip ci] on default branch
  • Create build for jenkinsci/winp on release ci server
  • [Update Center] HTTP/404 on /current/updates/*.json* links
  • Add monitoring for CD secrets updates
  • Add .war.asc to get.jenkins.io
  • Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)
  • [INFRA-3046] Monitor Jenkins mirrors Age

• Issues added to the next milestone:

  • Hosting of CSP compatibility “microsite” repo
  • Spot reclamation caused build failures in core build