Infrastructure Team Meeting - August 26, 2025

[dduportal (Damien Duportal)]

2025 08 26 Jenkins Infra Meeting

Attendees

• @smerle33 (Stéphane Merle)

Announcements

1. Jenkins Weekly Releases
   • Last Week: 2.524 had some hiccups with timestamp see here and postmortem
   • This Week: 2.525
2. Announcements:
   • Last LTS: 2.516.2 went well so @dduportal work on packaging was perfect

Upcoming Calendar

• Next Weekly: 2.526
• Next LTS: 2.516.3 September 17, 2025
• Next Security Release as per jenkinsci-advisories:
• Upcoming credentials expirations (~3 weeks):
  • 31 August 2025 Netlify [infra.ci.jenkins.io] Netlify token expires the 31 August 2025 · Issue #4760 · jenkins-infra/helpdesk · GitHub
  • 01 September 2025 NPM [infra.ci.jenkins.io] NPM token expires the 01 September 2025 · Issue #4766 · jenkins-infra/helpdesk · GitHub
  • 10 September 2025 [trusted.ci] Renew Cloudflare R2 Bucket Credential
  • Files Share
    • 08 September 2025 contributors.jenkins.io [azure] New end date for `contributors.jenkins.io` File Share service principal writer on `infra.ci.jenkins.io` (current: "2025-09-08T00:00:00Z") by jenkins-infra-updatecli[bot] · Pull Request #1167 · jenkins-infra/azure · GitHub
    • 08 September 2025 plugins.jenkins.io [azure] New end date for `plugins.jenkins.io` File Share service principal writer on `infra.ci.jenkins.io` (current: "2025-09-08T00:00:00Z") by jenkins-infra-updatecli[bot] · Pull Request #1168 · jenkins-infra/azure · GitHub
    • 08 September 2025 stats.jenkins.io [azure] New end date for `stats.jenkins.io` File Share service principal writer on `infra.ci.jenkins.io` (current: "2025-09-08T00:00:00Z") by jenkins-infra-updatecli[bot] · Pull Request #1169 · jenkins-infra/azure · GitHub

Cloud Budgets

• Azure CDF - Remaining: ~$27k for 2025:

  • May: $3.5k (invoice)
  • June: $3,474 (invoice)
  • July: $4,289 (expecting invoice at ~4.4k)
  • August: $4,666 (forecast at $5,992.20, far bellow 6.4k monthly until end of 2025)
    • Efforts around decreasing costs are paying off: forecast went from 6.8k 2 weeks ago to 6k today.

• Azure Sponsorship (Microsoft Credits) - Remaining: $683 until 31 August 2025

  • May: $14.6k
  • June: $10.5k
  • July: $5,1k
  • August: $360
    • Fully decommissioned

• DigitalOcean - Remaining $13,189.16 until January 02, 2026

  • May: $310 (invoice)
  • June: $252 (invoice)
  • July: $196 (invoice)
  • August: $192 (forecast at $238)
    • Slight increase due to a training VM (expected)

• AWS:

  • CloudBees:

    • May: $548
    • June: $606
    • July: $635,75
    • August: $524 (forecast at $666)
      • Recent checks show increased data outbound on pkg.origin.origin.io

  • Sponsored account ($89,470 credits lefts until 2027 => ~17 months remaining)

    • May: $2.96
    • June: $367.47
    • July: $4,899.99
    • August: $5180 (forecast at $6k5 )
      • EC2 plugin started to shows “over-provisioning” behavior. WiP on fine tuning (GC orphaned, fallback to ondemand) to improve behavior [ci.jenkins.io] Pipelines are stuck in RPU (Agents slow to allocate / build timeouts due to agent reclaimed) · Issue #4768 · jenkins-infra/helpdesk · GitHub

• Jfrog Artifactory Usage

  • Storage: 1.29TB (+0.01TB)
  • Bandwidth:
    • May: 23.93 TB (22.21 Tb for ‘releases’ repository)
    • June: 22.94 TB (21.44 Tb for ‘releases’ repository)
    • July: 24.96 Tb (23.45 Tb for ‘releases’ repository)
    • August: 16,94 Tb - (14.92 Tb for ‘releases’ repository) - forecast at 21 Tb
      • Visible decrease since August 19th
      • Waiting from JFrog about robots.txt to prevent crawling

Notes

• Done:

  • Keep platform up to date:
    • [ci.jenkins.io/AWS] Bump aws provider to new major 6.x version
      • not so easy lots of dependency update to group
    • Update ci.jenkins.io, trusted.ci, cert.ci and release.ci to latest LTS version 2.516.2
  • Keep platform sane and maintainable:
    • Deprecate the Docker Image jenkinsciinfra/crond
    • [infra.ci.jenkins.io] Switch to a private container registry for images built in infra.ci and consumed in Azure
  • Support
    • ci.jenkins.io does not allocate an agent with the maven-bom label

• Work in Progress:

  • Keep platform up to date:
    • Update Jira LTS from 9.12.x to 10.3.x
  • Keep platform sane and maintainable:
    • Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)
    • Tombstone Puppet (and replace it by something else)
    • [stats.jenkins.io/infra-statistics] Move “data for the usage stats site” generation (from anonymized data) out from Andrew machine
    • [ci.jenkins.io] Monitor and Garbage collect data volume of the DockerHub registry mirror and EC pull through cache
      • Added a BLOBDESCRIPTOR as per pull through cache (proxy): docker pull error "unexpected EOF" due to common blob expiration · Issue #2367 · distribution/distribution · GitHub to disable the blob descriptor cache
    • Install CSP plugin on trusted.ci, infra.ci, etc.
    • [trusted.ci.jenkins.io/updates.jenkins.io] Decrease costs generated by the Update Center generation
    • 2025 Cloud Usage: ensure that we can run until end of year
  • Support:
    • sg.mirror.servanamanaged.com mirror not listed in Mirrorstats #4782
      • one of the servanamanaged.com mirror seems out of the list … added it back
    • [ci.jenkins.io] Pipelines are stuck in RPU
      • could be close ?
    • [ci.jenkins.io] diagnose and eventually fix the EC2 Windows VM performances and build failures/retries
    • Admin access for Jenkins GSoC org admins to GSoC SiG Gitter channel
    • Github Copilot organisation space
    • ci.jenkins.io pages are slow to respond
    • [release.ci.jenkins.io/trusted.ci.jenkins.io] Ensure Core Package build only copy package indexes/websites to pkg.origin.jenkins.io VM
    • usage-in-plugins doesn’t run daily
  • Azure CDF Billing to decrease (2025 Cloud Usage: ensure that we can run until end of year):
    • [Azure] Merge webservices data storage accounts into a single one with NFS v4.1
    • [pkg.jenkins.io] migrate the pkg.origin.jenkins.io service from AWS VM to Azure publick8s
  • JFrog Artifactory (repo.jenkins-ci.org) outbound bandwitdh decrease:
    • Reduce artifactory bandwidth used by ci.jenkins.io
    • Reduce artifactory bandwidth used by infra.ci.jenkins.io

• Issues staying in backlog/triage:

  • [infra.ci.jenkins.io] NPM token expires the 01 September 2025
  • [updates.jenkins.io] set up mirrorbits to keep serving update-center from mirrors even if outdated
  • [infra.ci.jenkins.io] Netlify token expires the 31 August 2025
  • private docker image registry for staging core security releases
  • [Azure]: credentials less Service Principal
  • Tracking Issue for Groovy Script Conversion in RPU
  • Migrate census.jenkins.io VM from AWS CloudBees to DigitalOcean
  • Chinese jenkins site incorrect site redirection
  • [Azure] Migrate (e.g. re-create) AKS clusters publick8s and privatek8s with modern settings (private API, Azure Linux, NAT outbound)
  • [cert.ci/trusted.ci/private.vpn] Default outbound access for VMs in Azure will be retired
  • [private.vpn.jenkins.io] Azure deprecates Public IPs of type “Basic” the 30 September 2025
  • Add a real-world job to weekly.ci.jenkins.io
  • Move collection of stats out from Kohsuke’s home
  • Support [skip ci] on default branch
  • Create build for jenkinsci/winp on release ci server
  • [Update Center] HTTP/404 on /current/updates/*.json* links
  • dnf5 update fails with gpgcheck=1
  • Add monitoring for CD secrets updates
  • [INFRA-3046] Monitor Jenkins mirrors Age

• Issues added to the next milestone: N.A.