Infrastructure Team Meeting - August 19, 2025

[dduportal (Damien Duportal)]

2025 08 19 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @MarkEWaite (Mark Waite)
• @poddingue (Bruno Verachten)
• @kmartens27 (Kevin Martens)
• @biru-codeastromer (Birajit)

Announcements

1. Jenkins Weekly Releases
   • Last Week: 2.523 - successful - You're invited to talk on Matrix
   • This Week: 2.524 - started on time 1h30 ago - You're invited to talk on Matrix
     • Note: watching carefully the packaging step due to the file system changes
2. Announcements:
   • Team off:
     • Damien is off next week (25 → 29)
     • Stephane is back next week (25)
   • Infra Roadmap
     • Jenkins Roadmap
     • Next Priorities:
       • Azure CDF costs (includes moving pkg.origin to Azure CDF - less data transfer!)
   • Reminder: LTS release tomorrow (see belox)

Upcoming Calendar

• Next Weekly: 2025-08-26 - 2.525
• Next LTS: 2025-08-20 (tomorrow!) - 2.516.2
  • Next baseline will be selected 2025-09-03
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • 2025-08-31: [infra.ci.jenkins.io] Netlify token expires the 31 August 2025
  • 2025-09-01: [infra.ci.jenkins.io] NPM token expires the 01 September 2025

Cloud Budgets

• Azure CDF - Remaining: $32,154 for 2025:

  • May: $3.5k (invoice)
  • June: $3,474 (invoice)
  • July: $4,289 (expecting invoice at ~4.4k)
  • August: $3,517 (forecast at ~6.5k , we can only afford 6.4k monthly until end of 2025)
    • Efforts around decreasing costs are paying off: forecast went from 6.8k last friday to 6.5k today.

• Azure Sponsorship (Microsoft Credits) - Remaining: $683 until 31 August 2025

  • May: $14.6k
  • June: $10.5k
  • July: $5,1k
  • August: $360
    • Fully decommissioned

• DigitalOcean - Remaining $13,242.16 until January 02, 2026

  • May: $310 (invoice)
  • June: $252 (invoice)
  • July: $196 (invoice)
  • August: $140 (forecast at $240)
    • Slight increase due to a training VM (expected)

• AWS:

  • CloudBees:

    • May: $548
    • June: $606
    • July: $635,75
    • August: $399 (forecast at $658)
      • Recent checks show increased data outbound on pkg.origin.origin.io

  • Sponsored account ($89,470 credits lefts until 2027 => ~17 months remaining)

    • May: $2.96
    • June: $367.47
    • July: $4,899.99
    • August: $3,776.28 (forecast at $6.0k )
      • EC2 plugin started to shows “over-provisioning” behavior. WiP on fine tuning (GC orphaned, fallback to ondemand) to improve behavior

• Jfrog Artifactory Usage

  • Storage: 1.28TB
  • Bandwidth:
    • May: 23.93 TB (22.21 Tb for ‘releases’ repository)
    • June: 22.94 TB (21.44 Tb for ‘releases’ repository)
    • July: 24.96 Tb (23.45 Tb for ‘releases’ repository)
    • August: 12.62 Tb - (11.14 Tb for ‘releases’ repository) - forecast at 20.6 Tb
      • Visible decrease since August 6th (compared to July)
      • But some days are twice the amount (Thursdays ~840Gb while mondays ~410 Gb)
      • Waiting from JFrog about robots.txt to prevent crawling

Notes

• Done:

  • Keep platform up to date:
    • [get.jenkins.io/updates.jenkins.io] Bump mirrorbits from 0.5.1 to 0.6.0
      • Required switching to NFS storage account
      • 0.6.1 already deployed (bugfix including using fallback when Redis is down)
  • Keep platform sane and maintainable:
    • OSUOSL: decrease mirror dependency
      • Sync to OSUOSL is run (async: cron job) every 30 min from archives.jenkins.io
    • Deprecate ircbot service
      • Thanks Kevin and Tim!
  • Support
    • Account removal
    • ATH runs much slower than usual
      • Legit issue, same as [ci.jenkins.io] Pipelines are stuck in RPU (Agents slow to allocate / build timeouts due to agent reclaimed) · Issue #4768 · jenkins-infra/helpdesk · GitHub

• Work in Progress:

  • Keep platform sane and maintainable:

    • Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)
      • We have our first private job monitoring in place for docker-404
      • No alert (yet) and no timestamp check (yet)
      • Jay is off for health reasons, will be back on this (with Stephane help if needed)
    • Tombstone Puppet (and replace it by something else)
      • On hold
    • [stats.jenkins.io/infra-statistics] Move “data for the usage stats site” generation (from anonymized data) out from Andrew machine
      • On hold
    • [ci.jenkins.io] Monitor and Garbage collect data volume of the DockerHub registry mirror and EC pull through cache
      • On hold

  • Support:

    • Pipelines are stuck in RPU
      • Waiting if today’s fine tuning of ec2 plugin works as expected (fallback ondemand)
    • [ci.jenkins.io] diagnose and eventually fix the EC2 Windows VM performances and build failures/retries
      • Stuck: need to determine example of current slowness to see if it is EC2 only or general to Windows builds
      • Comparison with Azure Windows?
      • Note: Windows 2025 introduces “Dev Drive”, new FS (different from NTFS) from Windows 11 with performance improvement (seen in Java build - Microsoft blog)
        • With Windows 2019 depreciation in March 2026, would be a good path
    • Admin access for Jenkins GSoC org admins to GSoC SiG Gitter channel
      • On hold
    • Github Copilot organisation space
      • On hold

  • Azure CDF Billing to decrease (2025 Cloud Usage: ensure that we can run until end of year):

    • [infra.ci.jenkins.io] Switch to a private container registry for images built in infra.ci and consumed in Azure
    • [Azure] Merge webservices data storage accounts into a single one with NFS v4.1
    • [pkg.jenkins.io] migrate the pkg.origin.jenkins.io service from AWS VM to Azure publick8s
      • No more OSUOSL filling from pkg
      • No more azcopy from pkg to Azure storage accounts
      • Next step: no more plugins/update center copied from/to pkg

  • JFrog Artifactory (repo.jenkins-ci.org) outbound bandwitdh decrease:

    • Reduce artifactory bandwidth used by ci.jenkins.io
      • Impressive result since 5 Aug.
      • We still have a quick win.: jenkinsci/docker has been identified as a huge consumer. PR in progress by Mark and Herve.
    • Reduce artifactory bandwidth used by infra.ci.jenkins.io
      • No work done yet, but is needed regarding the Azure CDF cost: all downloads are going through the NAT gateway (so we pay for them)

  • Keep platform up to date:

    • Update Jira LTS from 9.12.x to 10.3.x
      • On hold (Messaged Ryan whom acknowledged)

• Issues staying in backlog/triage:

  • [infra.ci.jenkins.io] NPM token expires the 01 September 2025
    • For next week when @smerle will be back
  • [infra.ci.jenkins.io] Netlify token expires the 31 August 2025
    • For next week when @smerle will be back
  • [updates.jenkins.io] set up mirrorbits to keep serving update-center from mirrors even if outdated
  • private docker image registry for staging core security releases
  • [Azure]: credentials less Service Principal
  • Tracking Issue for Groovy Script Conversion in RPU
  • Migrate census.jenkins.io VM from AWS CloudBees to DigitalOcean
  • Chinese jenkins site incorrect site redirection
  • [Azure] Migrate (e.g. re-create) AKS clusters publick8s and privatek8s with modern settings (private API, Azure Linux, NAT outbound)
  • [cert.ci/trusted.ci/private.vpn] Default outbound access for VMs in Azure will be retired
  • [private.vpn.jenkins.io] Azure deprecates Public IPs of type “Basic” the 30 September 2025
  • Add a real-world job to weekly.ci.jenkins.io
  • Move collection of stats out from Kohsuke’s home
  • Support [skip ci] on default branch
  • Create build for jenkinsci/winp on release ci server
  • [Update Center] HTTP/404 on /current/updates/*.json* links
  • dnf5 update fails with gpgcheck=1
  • Add monitoring for CD secrets updates
  • [INFRA-3046] Monitor Jenkins mirrors Age

• Issues added to the next milestone:

  • usage-in-plugins doesn’t run daily