Infrastructure Team Meeting - November 25, 2025

[dduportal (Damien Duportal)]

2025 11 25 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @poddingue (Bruno Verachten)
• @MarkEWaite (Mark Waite)

Announcements

1. Jenkins Weekly Releases
   • Last Week:
     • 2025-11-18 - 2.537 - successful - You're invited to talk on Matrix
       • Introduced a new major bug with SSH agents. See 2.538 below.
       • Unified RPMs for Redhat and openSUSE required a few hotfixes to ensure completion and caused https://issues.jenkins.io/browse/JENKINS-76296
     • 2025-11-19 - 2.538 - successful - You're invited to talk on Matrix
       • Unplanned weekly release fixing the SSH bug. See Changelog.
   • This Week: 2.539 - started on time - You're invited to talk on Matrix
     • Packaging will generate all packages on release.ci.jenkins.io instead of pkg.origin VM for the first time
     • Contains a CSP bugfix
     • Release process failed (test issues), re-triggered
2. Announcements:
   • Migration JIRA → GitHub issues for Jenkins Core
     • See Jenkins core issues are migrating from Jira to GitHub | Jenkins Status Page
   • Weekly trigger time: What if we trigger weeklies 6 hours earlier to fit with Jay’s timezone?
     • => Jay and Mark not available, let’s delay this discussion next week
   • Infra Roadmap: Jenkins Roadmap
     • Current Priority: pkg.origin.jenkins.io migration
     • Roadmap deserves an update for beginning December (Herve + Damien)
     • Upcoming things not tracked by issues yet:
       • Docker 29.x => need an issue, but since docker-plugin fails with this version let’s put it on hold until fixed
       • Terraform 1.13 => need an issue
   • Jenkins Elections in progress. Check your emails for a voting link (only for Documentation officer)

Upcoming Calendar

• Next Weekly: 2025-12-02 - 2.540
• Next LTS: 2025-12-10 - 2.528.3 - Kris Stern is release lead, shadowed by Shalini Sudarsan, a newcomer \o/
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks): TBD by Damien for next milestone (next one is the 4th December)
• Next major event:
  • Contributor Summit + FOSDEM in Brussels, 30/31 Jan and 1st Feb
    • We have a booth for Jenkins at FOSDEM
    • Agenda for Summit is still WiP. Bring ideas (in community.jenkins.io)
    • Jenkins Contributor Summit on Jan 30, 2026 - Call for topics and ideas

Cloud Budgets

• Azure CDF - Remaining: ~$14.8k (31 Oct.) for 2025 - monthly threshold set at $6.0k

  • September: $5.8k (invoice)
  • October: $5.5k (waiting for final invoice)
  • November: $4.7k (forecast at 5.8k)
    • Huge spike on cert.ci.jenkins.io 2 weeks ago, but contained

• DigitalOcean - Remaining $11,451 until January 02, 2026 (13 month left at current rate)

  • September: $365 (invoice)
  • October: $788.81 (invoice)
  • November: $684 (forecast at $855)
    • Damien still to contact DigitalOcean for sponsoring renewal

• AWS:

  • CloudBees:

    • September: $709
    • October: $580
    • November: $438 (forecast at ~$530)

  • Sponsored account - $71,161 left (31 October 2025) until 2027 ($11,161.28 until 31 Jan. 2027 and $60,000 until 31 May 2027)=> ~11 months remaining at this rate

    • September: $5.4k
    • October: $6.9k
    • November: ~$4.7k (forecast at $5.8k)

• Jfrog Artifactory Usage

  • Storage: 1.49TB - slight increase
  • Bandwidth:
    • September: 19.31 Tb
    • October: 16.67 Tb
    • November: ~13.55 Tb - (forecast at 16.9 Tb)

Notes

• Done:

  • Support:
    • Plugin site builds failing on infra
      • GitHub transient issues caused the UC to fail, JS process didn’t followed redirect to the fallack, resolved itself
    • My change of email adress on jenkins.io/account hasn’t been synched to Jira
      • Known issue since at least 9 years, won’t fix (1 to 2 notices by users per year)
    • Add Daniel Krämer to @jenkins-infra/board
    • Request to Create New Repository for GSoC 2025 Project: Jenkins Domain Specific LLM
  • Keep Jenkins infrastructure sane and maintainable:
    • [Jenkins Controllers/JCasC] Warning message Configuration as Code obsolete file format: 'git' is obsolete, please use 'gitSource'
      • Finished last week by Jay
    • New mirror from Digital Shape Technologies in Canada
      • New mirror up and running

• Work in Progress:

  • [pkg.jenkins.io] migrate the pkg.origin.jenkins.io service from AWS VM to Azure publick8s

    • [release.ci.jenkins.io/trusted.ci.jenkins.io] Ensure Core Package build only copy package indexes/websites to pkg.origin.jenkins.io VM
      • dnf5 update fails with gpgcheck=1
        • Waiting for LTS to get the change (January)
      • https://get.jenkins.io/war-stable/latest/ does not point to latest stable release
        • Solved today with 2.539, waiting for 2.528.3 LTS to close
    • Everything detailled in the helpdesk issue above
    • End of a 12 years process \o/
    • Next steps: pkg.origin in Azure + improve the staging for upcoming LTS

  • Keep Jenkins infrastructure up to date:

    • Upgrade to Kubernetes 1.33
      • Initial agenda postponed to ensure both Herve and Jay have all tooling and accesses
      • Since Jay is unexpectdly unavailable, will wait for him to be back for a new agenda
      • Herve is checking his accesses. AKS is good (at least in read-only), EKS to do, tooling /runbook to finish
      • Damien to grant more accesses in Azure (Azure Group + permissions on some subscriptions but not all)
    • Add support for Windows 2025 agents
      • Docker Desktop installed and executed on Windows Server 2025
        • Windows 2022 and Windows 2025 run on it. Same as Windows Docker CE standalone (without Docker Desktop).
        • Startup is manual through UI, need effort to automate on headless => back to Docker CE Windows Engine.
        • Can run Linux container on this machine on WSL! Good to know for later.
        • Next steps:
          • Windows 2019 deprecation to be discussed with platform SIG.
          • For infra: let’s replace 2022 by 2025 first, then set it as default (and finally remove 2019 later once support stopped).
          • The AKS cluster privatek8s node pool need to be migrated out of Windows 2019 (both hosts and packaging image)

  • Migration of issues from JIRA to GitHub:

    • Migrate core components issues to GitHub
    • Migrate core issues to GitHub
    • Tim is managing the import, nothing on our side

  • Support:

    • [incrementals.jenkins.io/ci.jenkins.io] Outage of incrementals due to releasebot user’s API token reset
      • On hold
    • github-jenkinsci-permissions-report.json hasn’t been updated since Sep 12
      • On hold
    • ci.jenkins.io pages are slow to respond
      • On hold
    • Admin access for Jenkins GSoC org admins to GSoC SiG Gitter channel
      • On hold
    • [stats.jenkins.io/infra-statistics] Move “data for the usage stats site” generation (from anonymized data) out from Andrew machine
      • On hold

• Issues staying in backlog/triage:

  • Inform of Ingress NGINX Retirement in March 2026
  • Automated process for a plugin maintainer to request migration from Jira to GitHub issues
  • [Azure Deprecation] Convert your OS disks to Standard SSD or Premium SSD before 8 September 2028
  • [Azure] Merge webservices data storage accounts into a single one with NFS v4.1
  • [updates.jenkins.io] set up mirrorbits to keep serving update-center from mirrors even if outdated
  • private docker image registry for staging core security releases
  • [Azure]: credentials less Service Principal
  • Tracking Issue for Groovy Script Conversion in RPU
  • Tombstone Puppet (and replace it by something else)
  • Chinese jenkins site incorrect site redirection
  • Add a real-world job to weekly.ci.jenkins.io
  • [ci.jenkins.io] Monitor and Garbage collect data volume of the DockerHub registry mirror and EC pull through cache
  • Move collection of stats out from Kohsuke’s home
  • Support [skip ci] on default branch
  • Create build for jenkinsci/winp on release ci server
  • [Update Center] HTTP/404 on /current/updates/*.json* links
  • Add monitoring for CD secrets updates
  • Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)
  • [INFRA-3046] Monitor Jenkins mirrors Age

• Issues added to the next milestone:

  • Setup a job to allow performing maintenance operations (cron, GCs of resources, etc.)
    • [get.jenkins.io,mirrors.updates.jenkins.io] Resume GeoIP database weekly update
    • Ensure the BOM cache filler runs successfully and is easier to use and monitor
      • Bom maintainers are confirming it works flawlessly
      • To be extended to cover backend-extension-indexer (which can already benefit from the same cache even if not exhausive)