Infrastructure Team Meeting - September 16, 2025

[dduportal (Damien Duportal)]

2025 09 16 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @jayfranco999 (Jay Reddy)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)

Announcements

1. Jenkins Weekly Releases
   • Last week: 2.527 did have an issue in packaging due to Artifactory metadata not updated. A fix has been applied to avoid the error to happen again: thanks Daniel for sharing the fix you use during security advisories!
   • This week: 2.528 tomorrow (see below)
     • Stephane and Jay takes care of status.jenkins.io (ci.jio will be offline). Damien follows up the advisory, Mark as backup in case of emergency.
     • weekly.ci will have to be updated, but no need to put it offline (no risk)
2. Announcements:
   • Slow down due to human changes in progress (no details yet)
   • Infra Roadmap
     • Jenkins Roadmap
     • Next Priority: usage.jenkins.io / pkg.origin migrations / Azure deprecations
       • publick8s (network setup)
       • trusted/VPN/cert.ci VMs (size and network)

Upcoming Calendar

• Next Weekly: 2025-09-17 (tomorrow - see below) - 2.528
• Next LTS: 2025-09-17 (tomorrow - see below) - 2.516.3
• Next Security Release as per jenkinsci-advisories: https://groups.google.com/g/jenkinsci-advisories/c/rOpnrQxJWEY
• Upcoming credentials expirations (~3 weeks): N.A. for September

Cloud Budgets

• Azure CDF - Remaining: ~$24k (31 Aug.) for 2025: - max Monthly threshold is now $6.0k

  • June: $3,474 (invoice)
  • July: $4,289 (invoice)
  • August: $5,816 (expecting invoice of $5.9k with support)
  • September: $3.6k (forecast at $6.4k )

• DigitalOcean - Remaining $12,986 until January 02, 2026

  • June: $252 (invoice)
  • July: $196 (invoice)
  • August: $226 (invoice)
  • September: $167.98 (forecast $296)
    • Slight increase due to the new Usage VM (expected)

• AWS:

  • CloudBees:

    • June: $606
    • July: $635.75
    • August: $662.80
    • September: $160 forecast at $616

  • Sponsored account (83,398.55 lefts until 2027 => ~16 months remaining)

    • June: $367.47
    • July: $4,899.99
    • August: $6,072.47
      • Caused by EC2 overprovisioning.
    • September: $1,274.86 (forecast $4,281.15)

• Jfrog Artifactory Usage

  • Storage: 1.35TB (+0.06TB)
    • Darin pointed we seem to be scraped by an external LLM / harvester system
  • Bandwidth:
    • June: 22.94 TB (21.44 Tb for ‘releases’ repository)
    • July: 24.96 Tb (23.45 Tb for ‘releases’ repository)
    • August: 21.09 Tb - (18.77 Tb for ‘releases’ repository)
    • September: ~11 Tb - (10.85 Tb for ‘releases’ repository)
      • Forecast at ~20Tb

Notes

• Done:

  • Support:
    • Delete my duplicate account
    • Help joining @jenkinsci org
    • create account failed
    • Install CSP plugin on trusted.ci, infra.ci, etc.
  • Keep platform sane and maintainable:
    • [ATH] make sure ATH build run on ondemand agent not spot as it cost too much time / money in retry
    • [private.vpn.jenkins.io] Azure deprecates Public IPs of type “Basic” the 30 September 2025

• Work in Progress:

  • Keep platform sane and maintainable:
    • [INFRA-1972] Migrate usage.jenkins.io VM from AWS CloudBees to DigitalOcean
      • VM is created and running (SSH trhough VPN, small disk)
      • WiP on Puppet role, then increase disk (to write the runbook)
      • Then we can work on stats
    • [Azure] Migrate (e.g. re-create) AKS clusters publick8s and privatek8s with modern settings (private API, Azure Linux, NAT outbound)
      • WiP on the new publick8s. Focus IPv6: NAT gateway in Azure do not support IPv6.
      • Important: public Ip change for services. Includes LDAP (but DNS).
    • [cert.ci/trusted.ci/private.vpn] Default outbound access for VMs in Azure will be retired
      • We can start as the VPN has a supported public IP now.
    • [pkg.jenkins.io] migrate the pkg.origin.jenkins.io service from AWS VM to Azure publick8s
      • On hold until security advisory is finished
      • Next step: packaging in the release.ci agent + persistent volume (instead of remotely on this VM). Ubuntu 18.04 → 24.04.
    • [stats.jenkins.io/infra-statistics] Move “data for the usage stats site” generation (from anonymized data) out from Andrew machine
      • On hold until usage.jenkins.io on DigitalOcean is ready
    • [Azure] Merge webservices data storage accounts into a single one with NFS v4.1
      • On hold until publick8s is re-created with modern setup
    • Reduce artifactory bandwidth used by ci.jenkins.io
      • On hold until Docker controller job is adapted (then we’ll re-evaluate consumption)
    • Reduce artifactory bandwidth used by infra.ci.jenkins.io
      • Closeable unless we have new data showing high consumption on this one
    • Tombstone Puppet (and replace it by something else)
      • On hold (most probably until mid-Oct.)
    • Update Jira LTS from 9.12.x to 10.3.x
      • @dduportal to resume work on the upcoming (was on hold)
    • [ci.jenkins.io] Monitor and Garbage collect data volume of the DockerHub registry mirror and EC pull through cache
      • On hold until new publick8s cluster is there
    • Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)
      • On hold: Jay is taking a technical training
      • Note: we need to find another solution for time check in the synthetic monitor
  • Keep platform up to date:
    • Add support for Windows 2025 agents
      • WiP: works alone, but requires pipeline to switch to scripted syntax (memory issue)
  • Support:
    • Failure to stage Jenkins weekly security fix
      • Daniel is on it: we’ll post mortem after the advisory
    • Request access to release.ci.jenkins.io for hlemeur
      • Should not be needed for tomorrow (security advisory)
      • Waiting for confirmation for the VPN part
    • Implement a retry mechanism with non-spot instance in jenkinsci/jenkins pipeline
      • Waiting for Herve (pipeline retry on Jenkins Core)
    • ci.jenkins.io pages are slow to respond
      • Require login for ci.jenkins.io, only show pages to users
      • Use weekly.ci.jenkins.io for demonstration, include a few interesting jobs on weekly.ci.jenkins.io
    • [ci.jenkins.io] Pipelines are stuck in RPU (Agents slow to allocate / build timeouts due to agent reclaimed)
      • Still there. On hold until we finished Azure depreciations.
    • Admin access for Jenkins GSoC org admins to GSoC SiG Gitter channel
      • No feedback from Elements support. @dduportal to ping them again.

• Issues staying in backlog/triage:

  • [updates.jenkins.io] set up mirrorbits to keep serving update-center from mirrors even if outdated
  • private docker image registry for staging core security releases
  • [Azure]: credentials less Service Principal
  • Tracking Issue for Groovy Script Conversion in RPU
  • Migrate census.jenkins.io VM from AWS CloudBees to DigitalOcean
  • Chinese jenkins site incorrect site redirection
  • Add a real-world job to weekly.ci.jenkins.io
  • Move collection of stats out from Kohsuke’s home
  • Support [skip ci] on default branch
  • Create build for jenkinsci/winp on release ci server
  • [Update Center] HTTP/404 on /current/updates/*.json* links
  • dnf5 update fails with gpgcheck=1
  • Add monitoring for CD secrets updates
  • [INFRA-3046] Monitor Jenkins mirrors Age

• Issues added to the next milestone:

  • Admin request access to ci.jenkins.io for hlemeur
  • ci.jenkins.io is struggling to allocate Windows agents