Infrastructure Team Meeting - October 28, 2025

[dduportal (Damien Duportal)]

2025 10 28 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @poddingue (Bruno Verachten)
• @hlemeur (Hervé Le Meur)
• @jayfranco999 (Jay Reddy)

Announcements

1. Jenkins Weekly Releases
   • Last Week: 2.533 - successful: You're invited to talk on Matrix
     • Had to retry 3 times (tests failures)
   • This Week: 2.534 - started on time: You're invited to talk on Matrix
2. Announcements:
   • Slow team:
     • Mark is off until the 10 November 2025
     • Stephane is not available today
     • Jay is focused on resting and training on Kubernetes
   • JDK Patche for Q4 2025
     • Some JDKs are available, but other are still not there
     • Issue to create for next week
   • Infra Roadmap: Jenkins Roadmap
     • Current Priority: census.jio and pkg.origin migrations / Azure deprecations

Upcoming Calendar

• Next Weekly: 2025-11-04 - 2.535
• Next LTS: 2.528.2
  • 2025-10-29 (tomorrow): RC
  • 2025-11-12: Herve Le Meur release lead
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • 2025-10-30: Expiration of the Digital Oceans PATs - [infra.ci.jenkins.io] 2025-10-30 Expiration of the Digital Oceans PATs used by Terraform · Issue #4849 · jenkins-infra/helpdesk · GitHub
  • 2025-11-05: Azure Credential used by infra.ci to build packer Azure VMs - [infra.ci.jenkins.io] Azure Credential used by Packer Image builds expires the 05 November 2025 · Issue #4850 · jenkins-infra/helpdesk · GitHub
• Next major event:
  • FOSDEM: 31 Jan. and 1st Feb. in Brussels with Contributor Summit the 30

Cloud Budgets

• Azure CDF - Remaining: ~$20k (30 Sept.) for 2025 - max Monthly threshold is $6.0k

  • July: $4,289 (invoice)
  • August: $5,816 (invoice)
  • September: $5.8k (invoice)
  • October: $4.8k (forecast at $5.5k)
    • This is really good!

• DigitalOcean - Remaining $12,105 until January 02, 2026

  • July: $196 (invoice)
  • August: $226 (invoice)
  • September: $365
  • October: $663 (forecast: $760)
    • Increased costs dues to usages.jenkins.io fully migrated (expected)

• AWS:

  • CloudBees:
    • July: $635.75
    • August: $662.80
    • September: $709,17
    • October: $514 (forecast at ~$570)
      • Decreased as we migrated usage.jenkins.io to DigitalOcean
  • Sponsored account - $78,051 left (30 September 2025) until 2027 ($18,051.23 until 31 Jan. 2027 and $60,000 until 31 May 2027)=> ~10 months remaining at this rate
    • July: $4,899.99
    • August: $6,072.47
    • September: $5,347.30
    • October: $6.0k (forecast at $6.8k)
      • Increased usage due to an increase in ci.jenkins.io workload AND agent allocations issues (and retries)
      • Too soon to measure the effect of disabling Spot instances on the daily billing

• Jfrog Artifactory Usage

  • Storage: 1.46TB - steady
  • Bandwidth:
    • July: 24.96 Tb
    • August: 21.09 Tb
    • September: 19.31 Tb
    • October: ~14.05 Tb - (forecast at 16.0 Tb)
      • Visible decrease! Less abuse \o/

Notes

• Done:
  • Support:
    • Spammer in Jira
    • CD GitHub Action failed with repo.jenkins-ci.org: status code: 500, reason phrase: (500)
  • Keep platform up to date:
    • Java 25 outdated on ci.jenkins.io (aka JDK25 from EA to GA)
  • Keep platform sane and maintainable:
    • [puppet.jenkins.io,trusted.ci.jenkins.io, cert.ci.jenkins.io] Decrease VM sizes to consume less
  • ci.jenkins.io (all closed in favor of the issues opened in the “work in progress”):
    • ci.jenkins.io timeout when allocating linux-amd64 agent
    • Agents (presumably randomly?) disconnect from core builds
• Closed as not planned:
  • Maven deployment 401 error for sonargraph-integration plugin
• Work in Progress:

  • ci.jenkins.io:

    • [ci.jenkins.io] Pipelines are stuck in RPU (Agents slow to allocate / build timeouts due to agent reclaimed)
      • No more spot oàn EC2 and distinct instance sizes for Linux/windows => should improve UX, but we have to check budget carefully
      • We still see issues with EC2 plugin (SSH host key problem)
    • Non spot windows agents disappearing
      • Need to look again after changes aboves
    • ci.jenkins.io pages are slow to respond
      • Need to resume review of Herve’s contribution

  • Keep platform up to date:

    • [infra.ci.jenkins.io] 2025-10-30 Expiration of the Digital Oceans PATs used by Terraform
    • [infra.ci.jenkins.io] Azure Credential used by Packer Image builds expires the 05 November 2025
    • Update Jira LTS from 9.12.x to 10.3.x
      • LF is ready for migration
      • They need a time window for the maintenance, 6 hours of downtime
      • @dduportal planning it with Ryan from LF

  • Keep platform sane and maintainable:

    • Migrate census.jenkins.io VM from AWS CloudBees to DigitalOcean
      • Need to create the VM in DigitalOcean, empty but setup to be a trusted.ci private agent
      • Priority: stop the AWS CloudBees machine
    • [pkg.jenkins.io] migrate the pkg.origin.jenkins.io service from AWS VM to Azure publick8s
      • Started to reproduce the packaging steps locally
      • Next step: run a dry run of packaging (without publication) only inside Azure
    • [ci.jenkins.io,cert.ci.jenkins.io, trusted.ci.jenkins.io] Jenkins JDK tools: rationalize
      • We have stopped donwload JDK installers on each build
      • trusted.ci only uses JDK21 now
      • permanent agent now verify downloads
      • Next step: add JDK25 on cert.ci and trusted.ci + cyrptographic checks on packer images
      • And then: verify the JDKs patches are delivered to all locations
    • Reduce permissions of jenkins-release-bot in jenkinsci-cert organization
      • Should be closeable, waiting for Daniel last checks
      • Underline the need to replace this “bot” (which is a GH technical user) by a GH App
    • [agent.trusted.ci.jenkins.io] Adapt disks to current usage and Azure recommandations
      • Delayed due to quotas + we need a new “zonal VM”
      • On hold

  • Keep platform up to date:

    • Enforce at least TLS 1.2 in our webservices

  • Support:

    • github-jenkinsci-permissions-report.json hasn’t been updated since Sep 12
      • On hold: need to provide “non spot” agents on infra.ci.jenkins.io
    • Admin access for Jenkins GSoC org admins to GSoC SiG Gitter channel
      • On hold: @dduportal need to fallback to the board
• Issues staying in backlog/triage:
  • https://get.jenkins.io/war-stable/latest/ does not point to latest stable release
  • [Azure Deprecation] Convert your OS disks to Standard SSD or Premium SSD before 8 September 2028
  • [release.ci.jenkins.io/trusted.ci.jenkins.io] Ensure Core Package build only copy package indexes/websites to pkg.origin.jenkins.io VM
  • [Azure] Merge webservices data storage accounts into a single one with NFS v4.1
  • [updates.jenkins.io] set up mirrorbits to keep serving update-center from mirrors even if outdated
  • private docker image registry for staging core security releases
  • [Azure]: credentials less Service Principal
  • Tracking Issue for Groovy Script Conversion in RPU
  • Tombstone Puppet (and replace it by something else)
  • Chinese jenkins site incorrect site redirection
  • Add a real-world job to weekly.ci.jenkins.io
  • [ci.jenkins.io] Monitor and Garbage collect data volume of the DockerHub registry mirror and EC pull through cache
  • Move collection of stats out from Kohsuke’s home
  • Support [skip ci] on default branch
  • Create build for jenkinsci/winp on release ci server
  • [Update Center] HTTP/404 on /current/updates/*.json* links
  • dnf5 update fails with gpgcheck=1
  • Add monitoring for CD secrets updates
  • Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)
  • [INFRA-3046] Monitor Jenkins mirrors Age
• Issues added to the next milestone:
  • [increments.jenkins.io/ci.jenkins.io] Outage of incrementals due to releasebot user’s API token reset
  • Upgrade to Kubernetes 1.33
  • [stats.jenkins.io/infra-statistics] Move “data for the usage stats site” generation (from anonymized data) out from Andrew machine
  • Add support for Windows 2025 agents