Infrastructure Team Meeting - August 12, 2025

[dduportal (Damien Duportal)]

2025 08 12 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @jayfranco999 (Jay Reddy)
• @MarkEWaite (Mark Waite)
• @kmartens27 (Kevin Martens)

Announcements

1. Jenkins Weekly Releases
   • Last Week: 2.522 - successful - You're invited to talk on Matrix
   • This Week: 2.523 - started on time - You're invited to talk on Matrix
2. Announcements:
   • Team off:
     • Friday 15 August is banking day in France, India, Spain, Belgium (at least!)
     • Stephane is off until end of August
     • Damien might be off 25 → 29 August
   • Infra Roadmap
     • Jenkins Roadmap
     • Next Priorities:
       • Azure CDF costs (includes moving pkg.origin to Azure CDF - less data transfer!)

Upcoming Calendar

• Next Weekly: 2025-08-19 - 2.524
• Next LTS: 2025-08-20 - 2.516.2
  • Next baseline will be selected 2025-09-03
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • 2025-08-31: [infra.ci.jenkins.io] Netlify token expires the 31 August 2025
  • 2025-09-01: [infra.ci.jenkins.io] NPM token expires the 01 September 2025

Cloud Budgets

• Azure CDF - Remaining: $32,154 for 2025:

  • May: $3.5k (invoice)
  • June: $3,474 (invoice)
  • July: $4,289 (expecting invoice at ~4.4k)
  • August: $2143 (forecast at ~6.9k , we can only afford 6.4k monthly until end of 2025)
    • Increase due to workload migrated last week (ephemeral VM agents for trusted,cert and infra, packer builds)
    • Need to quickly analyse and act

• Azure Sponsorship (Microsoft Credits) - Remaining: $683 until 31 August 2025

  • May: $14.6k
  • June: $10.5k
  • July: $5,1k
  • August: $354
    • Fully decommissioned: we have ~$650 left for a few days (will use for ci.jenkins Windows VM agent slowness diagnostics)

• DigitalOcean - Remaining $13,343.55 until January 02, 2026

  • May: $310 (invoice)
  • June: $252 (invoice)
  • July: $196 (invoice)
  • August: $69 (forecast at $195)

• AWS:

  • CloudBees:

    • May: $548
    • June: $606
    • July: $635,75
    • August: $208 (forecast at $602)
      • Recent checks show increased data outbound on pkg.origin.origin.io

  • Sponsored account ($89,470 credits lefts until 2027 => ~17 months remaining)

    • May: $2.96
    • June: $367.47
    • July: $4,899.99
    • August: $2,190 (forecast at $5.5k)

• Jfrog Artifactory Usage

  • Storage: 1.24TB (slight increase of 10 Gb due to logs)
  • Bandwidth:
    • May: 23.93 TB (22.21 Tb for ‘releases’ repository)
    • June: 22.94 TB (21.44 Tb for ‘releases’ repository)
    • July: 24.96 Tb (23.45 Tb for ‘releases’ repository)
    • August: 7.68 Tb - (6.77 Tb for ‘releases’ repository) - forecast at 20 Tb
      • Visible decrease in the past 7 days

Notes

• Done:

  • Keep platform sane and maintainable:
    • [Update Center] Enforce HTTPS for updates.jenkins.io and updates.jenkins-ci.org
  • Decommission Azure Sponsored Subscription:
    • [get.jenkins.io/updates.jenkins.io] Move the managed redis-public instance out of Azure (sponsored) subscriptions
    • [Azure] Decommision sponsored subscription
  • Support:
    • Inconsistent labels result in missing JDK 21 installers
      • We ought to stop using Jenkins tool for JDK and Maven in the future
    • Mirror mirrors.de.sahilister.net - Network is unreachable
    • GSoC 2025 - Plugin Modernizer improvement - New repo for metadata-plugin-modernizer
    • Artifactory Not Caching Google’s Maven Repository
  • Keep platform up to date:
    • [infra.ci.jenkins.io/packer-images] Azure Credential used by Packer Image builds expires the 18 August 2025

• Work in Progress:

  • Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)
    • Pipeline Library works well
    • WiP on Datadog synthetics: PR opened, first wave of feedback
      • Might be delayed or slowed down as Jay does not have access to Datadog yet, but soon!
  • Azure CDF costs:
    • 2025 Cloud Usage: ensure that we can run until end of year
      • Need to open issues for trusted.ci, cert.ci, vpn and permanent agent
      • Biggest source of cost on Azure CDF: infra.ci data transfer.
        • Docker images in private registry?
        • infra.ci: decrease amount of packer builds and outbound data of these builds
    • [get.jenkins.io/updates.jenkins.io] Bump mirrorbits from 0.5.1 to 0.6.0
      • Hide the “merge storage volume for updates.jio, get.jio and many others” => perf & cost
    • OSUOSL: decrease mirror dependency
  • Keep platform up to date:
    • Update Jira LTS from 9.12.x to 10.3.x
      • Damien late to help and notify Ryan
  • Keep platform sane and maintainable:
    • Reduce artifactory bandwidth used by ci.jenkins.io
      • We saw improvements the first day. Waiting for confirmation (or more details if still consuming).
    • [ci.jenkins.io] diagnose and eventually fix the EC2 Windows VM performances and build failures/retries
      • Some improvements on Windows. But need help if we want things to improve.
      • We need to re-state “what is slow exactly” by comparing.
    • Reduce artifactory bandwidth used by infra.ci.jenkins.io
      • Amount of data seems low. To be confirmed (and closed), or updated
    • Tombstone Puppet (and replace it by something else)
      • Delayed to September, Haven’t heard back from Jack
    • [stats.jenkins.io/infra-statistics] Move “data for the usage stats site” generation (from anonymized data) out from Andrew machine
      • Damien started to work on this: VM in DigitlaOcean to create as code (now the PoC seems ok)
    • Deprecate ircbot service
      • Need help from Tim and the docs SIG
      • Damien to comment and ping them in the issue
    • [ci.jenkins.io] Monitor and Garbage collect data volume of the DockerHub registry mirror and EC pull through cache
      • Have not seen problem anymore: the “fix” worked as expected
      • Next step: implement a “cron” in infra.ci to clean up (same as the maven cache generation)
  • Support (both of them blocked):
    • Admin access for Jenkins GSoC org admins to GSoC SiG Gitter channel
      • Damien to follow Gavin email directions
    • Github Copilot organisation space
      • No news from GitHub

• Issues staying in backlog/triage:

  • [Azure] Migrate (e.g. re-create) AKS clusters publick8s and privatek8s with modern settings (private API, Azure Linux, NAT outbound)
  • [infra.ci.jenkins.io] Netlify token expires the 31 August 2025
  • private docker image registry for staging core security releases
  • [Azure]: credentials less Service Principal
  • Tracking Issue for Groovy Script Conversion in RPU
  • Migrate census.jenkins.io VM from AWS CloudBees to DigitalOcean
  • Chinese jenkins site incorrect site redirection
  • Add a real-world job to weekly.ci.jenkins.io
  • Move collection of stats out from Kohsuke’s home
  • Support [skip ci] on default branch
  • Create build for jenkinsci/winp on release ci server
  • [Update Center] HTTP/404 on /current/updates/*.json* links
  • dnf5 update fails with gpgcheck=1
  • Add monitoring for CD secrets updates
  • [INFRA-3046] Monitor Jenkins mirrors Age

• Issues added to the next milestone:

  • [updates.jenkins.io] set up mirrorbits to keep serving update-center from mirrors even if outdated
  • [cert.ci/trusted.ci/private.vpn] Default outbound access for VMs in Azure will be retired
  • [private.vpn.jenkins.io] Azure deprecates Public IPs of type “Basic” the 30 September 2025