Infrastructure Team Meeting - October 22, 2025

[dduportal (Damien Duportal)]

2025 10 22 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @smerle33 (Stéphane Merle)

Announcements

1. Jenkins Weekly Releases
   • 2.531 - successful: You're invited to talk on Matrix
   • 2.532 - successful: You're invited to talk on Matrix
   • This Week: 2.533 - successful: You're invited to talk on Matrix
     • Had to retry 3 times (tests failures)
2. Announcements:
   • Jenkins LTS 2.528.1 was released last week (on 15 Oct. 2025)
     • See You're invited to talk on Matrix
   • Infra Roadmap: Jenkins Roadmap
     • Current Priority: census.jio and pkg.origin migrations / Azure deprecations

Upcoming Calendar

• Next Weekly: 2025-10-28 - 2.534
• Next LTS:
  • 2025-10-29 - 2.528.2-rc
  • 2025-11-12 - 2.528.2 - Herve Le Meur release lead
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • 2025-10-30: Expiration of the Digital Oceans PATs
    • Issue to create (previous one: [infra.ci.jenkins.io] 2025-08-08 Expiration of the Digital Oceans PATs used by Terraform · Issue #4740 · jenkins-infra/helpdesk · GitHub)
  • 2025-11-05: Azure Credential used by infra.ci to build packer Azure VMs
    • Issue to create (previous one: [infra.ci.jenkins.io/packer-images] Azure Credential used by Packer Image builds expires the 18 August 2025 · Issue #4761 · jenkins-infra/helpdesk · GitHub)
• Next major event:
  • FOSDEM: 31 Jan. and 1st Feb. in Brussels

Cloud Budgets

• Azure CDF - Remaining: ~$20k (30 Sept.) for 2025 - max Monthly threshold is still at $6.0k ideally

  • July: $4,289 (invoice)
  • August: $5,816 (expecting invoice of $5.9k with support)
  • September: $5.8k (invoice)
  • October: $3.8k (forecast at $5.6k)

• DigitalOcean - Remaining $12,328.88 until January 02, 2026

  • July: $196 (invoice)
  • August: $226 (invoice)
  • September: $365
  • October: $439.87 (forecast: $650)

• AWS:

  • CloudBees:

    • July: $635.75
    • August: $662.80
    • September: $709,17
    • October: ~$387 (forecast at ~$568)
      • Decrease as we migrated usage.jenkins.io

  • Sponsored account - $78,051.23 left until 2027 ($18,051.23 until 31 Jan. 2027 and $60,000 until 31 May 2027)=> ~13 months remaining at this rate

    • July: $4,899.99
    • August: $6,072.47
    • September: $5,347.30
    • October: $4.8k (forecast at $6.8k )

• Jfrog Artifactory Usage

  • Storage: 1.46TB - steady
  • Bandwidth:
    • July: 24.96 Tb
    • August: 21.09 Tb
    • September: 19.31 Tb
    • October: ~10.92 Tb - (forecast at 16.5 Tb)

Notes

• Done:

  • Keep platform up to date:
    • [release.ci.jenkins.io] Azure Credentials for Core Release (Vault access) expires the 2025-10-29
      • Candidate for credential-less (see associated issue)
    • Update ci.jenkins.io, trusted.ci, cert.ci and release.ci to latest LTS version 2.528.1
  • Keep platform sane and maintainable:
    • [archives.jenkins.io] Data disk passed the 80% usage threshold
    • [release.ci.jenkins.io] Data disk passed the 80% usage threshold: need cleanup and/or size increase
    • Adapt JCasC config to ec2 2036.vb_105a_db_e4a_ea_ breaking change
    • Tombstone the “Bastion/Bounce” VM bounce.trusted.ci.jenkins.io
  • ci.jenkins.io:
    • ci.jenkins.io is unable to allocate agents
    • Timeout when building BOM
    • Implement a retry mechanism with non-spot instance in jenkinsci/jenkins pipeline
  • Support:
    • 2025-10-07 Outage (updates.jenkins.io, get.jenkins.io, and other services in publick8s)
      • Did not happened a 3rd time. Sounds like Azure network issue but no proof. Nothing obvious or no explicit error on our setup.
      • If it happens again: let’s try to add a NAT gateway on this cluster’s subnet to see if it fix the problem for IPv4 at least
    • Incrementals is not answering correctly
      • Last for 24h => clearly we have monitoring that could be improved
      • Caused by LTS upgrade on ci.jenkins.io (or correlated at least)
      • Caused by expired/invalid Jenkins User token: added rotation of this token in our routines
      • Maybe incremental could access ci.jenkins.io filesystem to retrieve the artifacts instead of using the Jenkins API
    • Invite lemeurherve to jenkins-infra GitHub organisation
    • Transient DNS resolution problem during BOM release
    • Symbolic link for “latest” for windows-stable in get.jenkins.io points to older release
    • get.jenkins.io root web page includes multiple dead and outdated links

• Work in Progress:

  • Keep platform sane and maintainable:
    • Migrate census.jenkins.io VM from AWS CloudBees to DigitalOcean
    • [agent.trusted.ci.jenkins.io] Adapt disks to current usage and Azure recommandations
      • Caused: [Azure Deprecation] Convert your OS disks to Standard SSD or Premium SSD before 8 September 2028
      • On hold: need a new Zonal VM and data migration cross AZs
  • Support:
    • Maven deployment 401 error for sonargraph-integration plugin
    • https://get.jenkins.io/war-stable/latest/ does not point to latest stable release
      • Links removed, need contribution on jenkinsci/packaging to add them back (same technique as for windows-stable / windows)
      • Back to triage
    • github-jenkinsci-permissions-report.json hasn’t been updated since Sep 12
      • All software problems solved
      • Failing many time due to spot agent being used. VM? Non spot?
    • Admin access for Jenkins GSoC org admins to GSoC SiG Gitter channel
      • No answer: let’s close
  • ci.jenkins.io:
    • Let’s disable spot for now and see the impact
    • Non spot windows agents disappearing
    • Agents (presumably randomly?) disconnect from core builds
    • ci.jenkins.io pages are slow to respond
    • [ci.jenkins.io] Pipelines are stuck in RPU (Agents slow to allocate / build timeouts due to agent reclaimed)
  • Keep platform up to date:
    • Java 25 outdated on ci.jenkins.io (aka JDK25 from EA to GA)
      • Let’s use GA from our packer images (for Jenkins tools) to avoid download on each build
    • Update Jira LTS from 9.12.x to 10.3.x
      • LF is working on the upgrade path. Almost there.
      • Damien to ping Ryan for status

• Issues staying in backlog/triage:

  • Proposed for “soon” (e.g. actionnables but not immediately or requires research):
    • Add support for Windows 2025 agents
      • Need to check if we can run 2019/2022 containers in Windows Server 2025 (HyperV isolation, Docker Desktop, other technique) like we do on Windows 11 machines
    • [Azure] Merge webservices data storage accounts into a single one with NFS v4.1
      • javadoc need tests
  • Reduce permissions of jenkins-release-bot in jenkinsci-cert organization
  • [updates.jenkins.io] set up mirrorbits to keep serving update-center from mirrors even if outdated
  • private docker image registry for staging core security releases
  • [Azure]: credentials less Service Principal
  • Tracking Issue for Groovy Script Conversion in RPU
  • Tombstone Puppet (and replace it by something else)
  • Chinese jenkins site incorrect site redirection
  • Add a real-world job to weekly.ci.jenkins.io
  • [ci.jenkins.io] Monitor and Garbage collect data volume of the DockerHub registry mirror and EC pull through cache
  • Move collection of stats out from Kohsuke’s home
  • Support [skip ci] on default branch
  • Create build for jenkinsci/winp on release ci server
  • [Update Center] HTTP/404 on /current/updates/*.json* links
  • dnf5 update fails with gpgcheck=1
  • Add monitoring for CD secrets updates
  • Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)
  • [INFRA-3046] Monitor Jenkins mirrors Age

• Issues added to the next milestone:

  • [pkg.jenkins.io] migrate the pkg.origin.jenkins.io service from AWS VM to Azure publick8s
    • Includes [release.ci.jenkins.io/trusted.ci.jenkins.io] Ensure Core Package build only copy package indexes/websites to pkg.origin.jenkins.io VM
  • Enforce at least TLS 1.2 in our webservices
  • [puppet.jenkins.io,trusted.ci.jenkins.io, cert.ci.jenkins.io] Decrease VM sizes to consume less
  • CD GitHub Action failed with repo.jenkins-ci.org: status code: 500, reason phrase: (500)
  • ci.jenkins.io timeout when allocating linux-amd64 agent
  • Upgrade to Kubernetes 1.33
    • Herve leads Jay on this one