Infrastructure Team Meeting - September 23, 2025

[dduportal (Damien Duportal)]

2025 09 23 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @jayfranco999 (Jay Reddy)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @hlemeur (Hervé Le Meur)

Announcements

1. Jenkins Weekly Releases
   • Last week: 2025-09-17 - 2.528 published with no issues as part of the security advisory
   • This Week: 2025-09-23 - 2.529 - (started on time, war file uploaded during the meeting) - (link to elements thread in jenkins/release channel to add)
     • Docker image publication has a new change which should only apply to ci.jio
2. Announcements:
   • Security Advisory last week (on 2025-09-17) went well. Core releases: 2.528 and 2.516.3
   • Still slow due to human changes in progress
   • Infra Roadmap
     • Jenkins Roadmap
     • Current Priority: usage.jenkins.io / pkg.origin migrations / Azure deprecations
       • publick8s (network setup)
       • trusted/VPN/cert.ci VMs (size and network)
   • Jenkins Elections voter registration started (see blog post )
   • FOSDEM dates announced: 31 Jan. and 1st Feb. in Brussels
     • We’ll start planning a Jenkins Contributor Summit!

Upcoming Calendar

• Next Weekly: 2025-09-30 - 2.530
• Next LTS: 2025-10-15 - 2.528.1 - (Hervé Le Meur is release lead, using release checklist)
  • 2025-10-01: Release Candidate
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • 2025-10-01 (see [azure] renewal of differents Service Principal Passwords · Issue #4808 · jenkins-infra/helpdesk · GitHub ):
    • [trusted.ci.jenkins.io] Azure SP used to deploy www.jenkins.io and javadoc.jenkins.io
    • [infra.ci.jenkins.io] Azure SP used for updatecli jobs and for docs.jenkins.io deployment
  • 2025-10-14 (see [trusted.ci.jenkins.io] RPU Artifactory API token expires the `2025-10-14` · Issue #4809 · jenkins-infra/helpdesk · GitHub ):
    • [trusted.ci.jenkins.io] Artifactory API token used by RPU builds expires

Cloud Budgets

• Azure CDF - Remaining: ~$24k (31 Aug.) for 2025: - max Monthly threshold is now $6.0k

  • June: $3,474 (invoice)
  • July: $4,289 (invoice)
  • August: $5,816 (expecting invoice of $5.9k with support)
  • September: $4.5k (forecast at $6.2k )
    • Caused by the publick8s migration

• DigitalOcean - Remaining $12,986 until January 02, 2026

  • June: $252 (invoice)
  • July: $196 (invoice)
  • August: $226 (invoice)
  • September: $251.10 (forecast $329)
    • Slight increase due to the new Usage VM (expected)

• AWS:

  • CloudBees:

    • June: $606
    • July: $635.75
    • August: $662.80
    • September: $475.32 forecast at $644.44

  • Sponsored account (83,398.55 lefts until 2027 => ~16 months remaining)

    • June: $367.47
    • July: $4,899.99
    • August: $6,072.47
      • Caused by EC2 overprovisioning.
    • September: $3,785.56 (forecast $5,296.81)

• Jfrog Artifactory Usage

  • Storage: 1.45TB (+0.1TB)
    • Darin adjusted logs, should have positive impact soon
  • Bandwidth:
    • June: 22.94 TB (21.44 Tb for ‘releases’ repository)
    • July: 24.96 Tb (23.45 Tb for ‘releases’ repository)
    • August: 21.09 Tb - (18.77 Tb for ‘releases’ repository)
    • September: ~14.9 Tb - (13.29 Tb for ‘releases’ repository)
      • Forecast at ~19.45Tb

Notes

• Done:

  • Support:
    • [ci.jenkins.io] Brownout disabling readonly access.
      • Outage on ci.jio => no more DDoS
      • Next brownout requires a bit of groovy/puppet setup on ci.jio to add the correct permissions to authenticated users (otherwise only admin have permissions)
    • PR-merge strategy used in Libraries
  • Keep platform up to date:
    • Update ci.jenkins.io, trusted.ci, cert.ci and release.ci to latest LTS version 2.516.3

• Work in Progress:

  • Keep platform sane and maintainable:
    • [Azure] Migrate (e.g. re-create) AKS clusters publick8s and privatek8s with modern settings (private API, Azure Linux, NAT outbound)
      • New cluster is created, and runs ~75% of applications
      • Remaining application to validate before migrating: LDAP “stack” , www.origin.jenkins.io (need a chart change), weekly.ci.jenkins.io
      • Planning full migration Wednesday, or Thursday (if last minute problems)
      • Outage on updates.jenkins.io and get.jenkins.io yesterday: @dduportal needs to write issue + post mortem.
        • Need to report issue on mirrorbits repo with logs
      • Kubernetes 1.32 will be next
    • [INFRA-1972] Migrate usage.jenkins.io VM from AWS CloudBees to DigitalOcean
      • VM recreated (due a minor terraform mistake we prefer fixing properly before migration)
        • VPN and firewall already updated. Might need minor updatecli manifest later
      • Data transfer from AWS to DO started (~147 Go). A bit more needed once the VM is back in shape
      • Next step: ensure Puppet provisioning works completely as expected (after VM recreation)
    • [stats.jenkins.io/infra-statistics] Move “data for the usage stats site” generation (from anonymized data) out from Andrew machine
      • On hold: require usage.jenkins.io to be migrated in DigitalOcean
    • [cert.ci/trusted.ci/private.vpn] Default outbound access for VMs in Azure will be retired
      • On hold: need publick8s to be migrated first
    • Reduce artifactory bandwidth used by ci.jenkins.io
      • jenkinsci/docker PR from Herve merged to use mirrors (get.jenkins.io) for WAR retrieval instead of Artifactory for ci.jenkins.io CI builds
        • We’ll confirm non regression for the publication today for the weekly release => trusted.ci is expected to retrieve the WAR file from Artifactory instead (so we can build Docker images even if packaging fails)
    • [pkg.jenkins.io] migrate the pkg.origin.jenkins.io service from AWS VM to Azure publick8s
      • On hold: due to publick8s migration AND the Artifactory reduction
    • [Azure] Merge webservices data storage accounts into a single one with NFS v4.1
      • Next candidate is www.jenkins.io
      • Part of the new publick8s cluster
    • [ci.jenkins.io] Monitor and Garbage collect data volume of the DockerHub registry mirror and EC pull through cache
      • On hold: (project “cron jobs” in infra.ci)
    • Implement a retry mechanism with non-spot instance in jenkinsci/jenkins pipeline
      • WiP by Hervé on this one. More tests to run.
      • On hold until other issues are fixed (see issue content)
      • Would be helpful/simpler to have Hervé with admin rights on ci.jenkins.io (see issue below)
    • Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)
      • On hold (Jay will resume after Kubernetes training and Jenkins laptop setup)
    • Reduce artifactory bandwidth used by infra.ci.jenkins.io
      • To be closed (no more data)
    • Tombstone Puppet (and replace it by something else)
      • Back to triage (no hands on deck here)
  • Keep platform up to date:
    • Add support for Windows 2025 agents
      • On hold (after the draft PR which seems to be good to roll)
      • Let’s merge the scripted pipeline PR! (need one last cleanup from Stephane, then we can review, merge and trigger a release)
      • Windows 2025 delayed (to avoid overcost)
    • Update Jira LTS from 9.12.x to 10.3.x
      • On hold until publick8s is migrated
  • ci.jenkins.io production issues:
    • [ci.jenkins.io] Pipelines are stuck in RPU (Agents slow to allocate / build timeouts due to agent reclaimed)
    • ci.jenkins.io pages are slow to respond
      • Initial try at a brownout failed (enabling login is OK, but missing permissions in groovy.d)
    • ci.jenkins.io agents are allocated but do not connect
      • Capacity issue is the root cause
      • But we also have EKS Terraform failures: need to investigate
  • Support:
    • Failure to stage Jenkins weekly security fix
    • Admin request access to ci.jenkins.io for hlemeur
      • On hold until new publick8s
    • Request access to release.ci.jenkins.io for hlemeur
      • On hold until new publick8s, needed for next LTS has he is release lead
    • Admin access for Jenkins GSoC org admins to GSoC SiG Gitter channel
      • No news, @dduportal to re-re-re-ping them

• Issues staying in backlog/triage:

  • [release.ci.jenkins.io/trusted.ci.jenkins.io] Ensure Core Package build only copy package indexes/websites to pkg.origin.jenkins.io VM
  • [updates.jenkins.io] set up mirrorbits to keep serving update-center from mirrors even if outdated
  • private docker image registry for staging core security releases
  • [Azure]: credentials less Service Principal
  • Tracking Issue for Groovy Script Conversion in RPU
  • Migrate census.jenkins.io VM from AWS CloudBees to DigitalOcean
  • Chinese jenkins site incorrect site redirection
  • Add a real-world job to weekly.ci.jenkins.io
  • Move collection of stats out from Kohsuke’s home
  • Support [skip ci] on default branch
  • Create build for jenkinsci/winp on release ci server
  • [Update Center] HTTP/404 on /current/updates/*.json* links
  • dnf5 update fails with gpgcheck=1
  • Add monitoring for CD secrets updates
  • [INFRA-3046] Monitor Jenkins mirrors Age

• Issues added to the next milestone:

  • [trusted.ci.jenkins.io] RPU Artifactory API token expires the 2025-10-14
  • [azure] renewal of differents Service Principal Passwords
  • ci.jenkins.io is struggling to allocate Windows agents