Infrastructure Team Meeting - September 02, 2025

[dduportal (Damien Duportal)]

2025 09 02 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @jayfranco999 (Jay Reddy)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @kmartens27 (Kevin Martens)
• @hlemeur (Hervé Le Meur)

Announcements

1. Jenkins Weekly Releases
   • Last Week: Success despite a packaging issue from @dduportal work 2 weeks ago. Fixed by @smerle and @MarkEWaite - 2.525
   • This Week: Started on time - 2.526
2. Announcements:
   • Infra Roadmap
     • Jenkins Roadmap
     • Next Priority: Azure CDF costs / pkg.origin to Azure CDF (less data transfer)
       • Kubernetes 1.32 (network setup)
       • trusted/VPN/cert.ci VMs (size and network)

Upcoming Calendar

• Next Weekly: 2025-09-07 - 2.527
• Next LTS: 2025-09-17 - 2.516.3 (Hervé le Meur is release lead)
  • RC for Wednesday. Build flaky (?) tests bypassed: incremental products.
• Next Security Release as per jenkinsci-advisories:
  • 2025-09-03 (tomorrow) - https://groups.google.com/g/jenkinsci-advisories/c/QPyjroxekAo
• Upcoming credentials expirations (~3 weeks):
  • 2025-09-08: New expiration date for the credential credentials_expiration_date in the cloudflare Terraform project
  • 2025-09-10:
    • [trusted.ci.jenkins.io] Update Center credential for Cloudflare R2 expires => @dduportal
    • All other Terraform (states) credentials: https://github.com/jenkins-infra/terraform-states/pulls => @dduportal to create issue, @smerle to run it

Cloud Budgets

• Azure CDF - Remaining: ~$27k for 2025:

  • May: $3.5k (invoice)
  • June: $3,474 (invoice)
  • July: $4,289 (invoice)
  • August: $5,816 (expecting invoice of $5.9k with support) - max threshold is $6.4k

• Azure Sponsorship (Microsoft Credits) - Fully decommissioned

• DigitalOcean - Remaining $13,148.63 until January 02, 2026

  • May: $310 (invoice)
  • June: $252 (invoice)
  • July: $196 (invoice)
  • August: $226 (invoice)
    • Slight increase due to a training VM (expected)

• AWS:

  • CloudBees:

    • May: $548
    • June: $606
    • July: $635.75
    • August: $662.80

  • Sponsored account ($89,470 credits lefts until 2027 => ~17 months remaining)

    • May: $2.96
    • June: $367.47
    • July: $4,899.99
    • August: $6,072.47 (a bit above the $6.0k expected threshold )
      • Caused by EC2 overprovisioning. Should be fixed for september: need to check the daily view
      • Spot vs. Data transfer challenge now.
      • And also : BOM optimization

• Jfrog Artifactory Usage

  • Storage: 1.29TB (+0.01TB)
  • Bandwidth:
    • May: 23.93 TB (22.21 Tb for ‘releases’ repository)
    • June: 22.94 TB (21.44 Tb for ‘releases’ repository)
    • July: 24.96 Tb (23.45 Tb for ‘releases’ repository)
    • August: 21.09 Tb - (18.77 Tb for ‘releases’ repository)
      • Visible decrease since August 19th
      • Waiting from JFrog about robots.txt to prevent crawling

Notes

• Done:

  • Keep platform up to date:
    • [[infra.ci.jenkins.io] Azure Service Principal Credentials]([infra.ci.jenkins.io] Azure Service Principal Credentials · Issue #4787 · jenkins-infra/helpdesk · GitHub
    • [infra.ci.jenkins.io] NPM token expires the 01 September 2025
    • [infra.ci.jenkins.io] Netlify token expires the 31 August 2025
  • Support:
    • [get.jenkins.io] 2 mirrors down
      • @dduportal to clean up the last mirror (and runbooks)
    • sg.mirror.servanamanaged.com mirror not listed in https://get.jenkins.io/index.html?mirrorstats
      • Left-overs of Redis + mirrorbits 0.6 migration
    • Reset password for hlemeur jenkins account
    • Failure to package weekly 2.525 on stage Get GPG key.
    • Plugin Health Scoring build on infra.ci.jenkins.io not passing

• Work in Progress:

  • ci.jenkins and AWS:
    • ci.jenkins.io pages are slow to respond
      • On hold to see if attacks come back. See issue for propsoed countermeasure.
    • [ci.jenkins.io] Pipelines are stuck in RPU
      • Ip problem solved
      • Fallback to ondemand fixed and working
      • Spot capacity again: new instance size? cross AZ? other,
        • We have the spot vs. cross AZ data transfer cost challenge now.
      • See Hervé issue around the allocation of nodes with retry to ondemand
    • [ci.jenkins.io] diagnose and eventually fix the EC2 Windows VM performances and build failures/retries
      • First hypothesis from James Nord: working around the Maven dependency resolution setup (which is slow on Windows)
      • Second hypothesis: Windows 2025 with DevDrive by default should help
      • Back to backlog (as we need to work on 2025 Server first)
    • [ci.jenkins.io] Monitor and Garbage collect data volume of the DockerHub registry mirror and EC pull through cache
      • On hold
  • Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)
    • WiP on the datadog synthetics, phase 3/3: compare last build timestamp to “now”.
      • Challenge: can the synthetic monitors use dynamic functions such as “now()”?
      • Need to think about alternatives
  • Azure CDF costs:
    • [pkg.jenkins.io] migrate the pkg.origin.jenkins.io service from AWS VM to Azure publick8s
    • [release.ci.jenkins.io/trusted.ci.jenkins.io] Ensure Core Package build only copy package indexes/websites to pkg.origin.jenkins.io VM
    • [trusted.ci.jenkins.io/updates.jenkins.io] Decrease costs generated by the Update Center generation
    • [Azure] Merge webservices data storage accounts into a single one with NFS v4.1
      • javadoc and www.jenkins.io are the next candidates
  • Artifactory Bandwidth Usage
    • Reduce artifactory bandwidth used by ci.jenkins.io
      • WiP on the ci.jenkins.io Docker image build (~800Go monthly)
    • Reduce artifactory bandwidth used by infra.ci.jenkins.io
      • On Hold, waiting for new data from Darin, putting back to backlog
  • Infra stats
    • [stats.jenkins.io/infra-statistics] Move “data for the usage stats site” generation (from anonymized data) out from Andrew machine
      • @smerle is going to take over and start the task of migrating usage.jenkins.io from AWS (CloudBees) to DigitalOcean, with enough capacity to also run stats on it.
  • Support:
    • Need access to private VPN for accessing infra.ci.jenkins.io,cert.ci.jenkins.io and ci.jenkins.io instances.
      • Waiting for user to confirm it work for them
    • Install CSP plugin on trusted.ci, infra.ci, etc.
      • cert.ci, infra.ci and weekly.ci are done
      • PR opened to add it to trusted.ci (involves a bit of puppet). Next is ci.
      • Todo: release.ci.
    • usage-in-plugins doesn’t run daily
      • On hold, back to backlog
    • Admin access for Jenkins GSoC org admins to GSoC SiG Gitter channel
      • On hold => @dduportal
    • Github Copilot organisation space
      • On hold, back to backlog
    • Tombstone Puppet (and replace it by something else)
      • @smerle is going to work on a PoC on packer-image to replace provisioning shells by Ansible (first step).
    • Update Jira LTS from 9.12.x to 10.3.x
      • On hold => @dduportal to resume

• Issues staying in backlog/triage::

  • [updates.jenkins.io] set up mirrorbits to keep serving update-center from mirrors even if outdated
  • private docker image registry for staging core security releases
  • [Azure]: credentials less Service Principal
  • Tracking Issue for Groovy Script Conversion in RPU
  • Migrate census.jenkins.io VM from AWS CloudBees to DigitalOcean
  • Chinese jenkins site incorrect site redirection
  • [Azure] Migrate (e.g. re-create) AKS clusters publick8s and privatek8s with modern settings (private API, Azure Linux, NAT outbound)
  • [cert.ci/trusted.ci/private.vpn] Default outbound access for VMs in Azure will be retired
  • [private.vpn.jenkins.io] Azure deprecates Public IPs of type “Basic” the 30 September 2025
  • Add a real-world job to weekly.ci.jenkins.io
  • Move collection of stats out from Kohsuke’s home
  • Support [skip ci] on default branch
  • Create build for jenkinsci/winp on release ci server
  • [Update Center] HTTP/404 on /current/updates/*.json* links
  • dnf5 update fails with gpgcheck=1
  • Add monitoring for CD secrets updates
  • [INFRA-3046] Monitor Jenkins mirrors Age

• Issues added to the next milestone:

  • [trusted.ci.jenkins.io] Update Center credential for Cloudflare R2 expires on 2025-09-10
  • Add support for Windows 2025 agents
  • Implement a retry mechanism with non-spot instance in jenkinsci/jenkins pipeline