Infrastructure Team Meeting - October 07, 2025

[dduportal (Damien Duportal)]

2025 10 07 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @smerle33 (Stéphane Merle)
• @hlemeur (Hervé Le Meur)

Announcements

1. Jenkins Weekly Releases
   • Last Week: 2.530 - started on time - no issues
     • You're invited to talk on Matrix
   • This Week: 2.531 - started on time
     • You're invited to talk on Matrix
2. Announcements:
   • Let’s cancel next week team meeting: Stephane will be alone not in days off
   • Still slow due to human changes
   • Outage today: see 2025-10-07 Outage (updates.jenkins.io, get.jenkins.io, and other services in `publick8s`) · Issue #4827 · jenkins-infra/helpdesk · GitHub
   • Infra Roadmap: Jenkins Roadmap
     • Current Priority: census.jio and pkg.origin migrations / Azure deprecations

Upcoming Calendar

• Next Weekly: 2025-10-14 - 2.532
• Next LTS: 2025-10-15 - 2.528.1 - Herve Le Meur is release lead
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • 2025-10-29: [release.ci.jenkins.io] Azure Credentials for Core Release (Vault)
    • Issue to create (previous one: [release.ci.jenkins.io] Azure Credentials for Core Release (Vault access) expires the `2025-08-10` · Issue #4741 · jenkins-infra/helpdesk · GitHub )
  • 2025-10-30: Expiration of the Digital Oceans PATs
    • Issue to create (previous one: [infra.ci.jenkins.io] 2025-08-08 Expiration of the Digital Oceans PATs used by Terraform · Issue #4740 · jenkins-infra/helpdesk · GitHub )
• Next major event:
  • FOSDEM: 31 Jan. and 1st Feb. in Brussels

Cloud Budgets

• Azure CDF - Remaining: ~$20k (30 Sept.) for 2025 - max Monthly threshold is still at $6.0k ideally

  • July: $4,289 (invoice)
  • August: $5,816 (expecting invoice of $5.9k with support)
  • September: $5.8k (forecast at $6.0k)
  • October: $1.2k (forecast at $5.3k)

• DigitalOcean - Remaining $12,686 until January 02, 2026

  • July: $196 (invoice)
  • August: $226 (invoice)
  • September: $365
  • October: $83 (forecast: $367)

• AWS:

  • CloudBees:

    • July: $635.75
    • August: $662.80
    • September: $709,17
    • October: $133,39 (forecast at $580)

  • Sponsored account - $78,051.23 left until 2027 ($18,051.23 until 31 Jan. 2027 and $60,000 until 31 May 2027)=> ~13 months remaining at this rate

    • July: $4,899.99
    • August: $6,072.47
    • September: $5,347.30
    • October: ~$1.4k (forecast at $6.2k )

• Jfrog Artifactory Usage

  • Storage: 1.46TB (slight increase - logs?)
  • Bandwidth:
    • July: 24.96 Tb
    • August: 21.09 Tb
    • September: 19.31 Tb
    • October: ~4.51 Tb - (forecast at 19.9 Tb)

Notes

• Done:

  • Keep platform sustainable:
    • [INFRA-1972] Migrate usage.jenkins.io VM from AWS CloudBees to DigitalOcean
    • [Azure Deprecation] Migrate general-purpose v1 storage accounts to v2 before 13 October 2026
  • Keep platform up-to-date:
    • Upgrade to Kubernetes 1.32
    • [trusted.ci.jenkins.io] RPU Artifactory API token expires the 2025-10-14
  • Support:
    • artifact-caching-proxy (http://artifact-caching-proxy.artifact-caching-proxy.svc.cluster.local:8080/): status code: 502, reason phrase: Bad Gateway (502)
    • BOM build false negative report to GitHub pull request

• Work in Progress:

  • Keep platform sustainable:

    • Migrate census.jenkins.io VM from AWS CloudBees to DigitalOcean
      • Need a team pairing session to understand what this VM does

  • Keep platform up-to-date:

    • Update Jira LTS from 9.12.x to 10.3.x
      • LDAP binding is good (confirmed by Ryan)
      • Initial tests.issues.jenkins.io (private) went well
      • WiP: Ryan and Mark are working on a migration case

  • Support:

    • 2025-10-07 Outage (updates.jenkins.io, get.jenkins.io, and other services in publick8s)
      • Still watching the behavior, but should be back to normal
      • Gotta close status.jenkins.io
      • Post mortem to write once Azure will have published their own status incident. Otherwise we’ll close with an half-post mortem end of milestone
    • github-jenkinsci-permissions-report.json hasn’t been updated since Sep 12
      • One last issue: use non spot agent for this one in particular
    • Support request: Testing Algolia integration for jenkins.io Vite.js migration
      • Related to the new docs.jenkins.io
      • @dduportal made answers in the system: there should be nothing else for infra
      • Moving to backlog
    • Failure to stage Jenkins weekly security fix
      • Moving to backlog (@dduportal will consult Daniel beck for other details)
    • Admin access for Jenkins GSoC org admins to GSoC SiG Gitter channel
      • No news from them, despite Gavin social posts. Let’s give one last ping or close the issue assuming we lost the channel.

  • ci.jenkins.io:

    • ci.jenkins.io pages are slow to respond
      • WiP: enabling mandatory authentication. Hervé is working on authorizations setup with the Groovy lockbox system. Initial PR just been opened.
    • [ci.jenkins.io] Pipelines are stuck in RPU (Agents slow to allocate / build timeouts due to agent reclaimed)
      • Back to triage as we need to rework the multi-AZ setup on AWS to solve this
    • Agents (presumably randomly?) disconnect from core builds
      • Todo: disable Spot instances on AWS for October and check the price impact
    • Implement a retry mechanism with non-spot instance in jenkinsci/jenkins pipeline
      • PR on jenkins/jenkinsci has been merged. Thanks Herve and Timja.
      • Sorry: we are going to disable Spot temporarily, but still usefull whatesoever!

• Issues staying in backlog/triage:

  • Upgrade to Kubernetes 1.33
  • Add support for Windows 2025 agents
  • [release.ci.jenkins.io/trusted.ci.jenkins.io] Ensure Core Package build only copy package indexes/websites to pkg.origin.jenkins.io VM
  • [Azure] Merge webservices data storage accounts into a single one with NFS v4.1
  • [updates.jenkins.io] set up mirrorbits to keep serving update-center from mirrors even if outdated
  • private docker image registry for staging core security releases
  • [Azure]: credentials less Service Principal
  • Tracking Issue for Groovy Script Conversion in RPU
  • Tombstone Puppet (and replace it by something else)
  • [stats.jenkins.io/infra-statistics] Move “data for the usage stats site” generation (from anonymized data) out from Andrew machine
  • Chinese jenkins site incorrect site redirection
  • Add a real-world job to weekly.ci.jenkins.io
  • [ci.jenkins.io] Monitor and Garbage collect data volume of the DockerHub registry mirror and EC pull through cache
  • Move collection of stats out from Kohsuke’s home
  • Support [skip ci] on default branch
  • Create build for jenkinsci/winp on release ci server
  • [Update Center] HTTP/404 on /current/updates/*.json* links
  • dnf5 update fails with gpgcheck=1
  • Add monitoring for CD secrets updates
  • [pkg.jenkins.io] migrate the pkg.origin.jenkins.io service from AWS VM to Azure publick8s
  • Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)
  • [INFRA-3046] Monitor Jenkins mirrors Age

• Issues added to the next milestone:

  • Transient DNS resolution problem during BOM release
  • Tombstone the “Bastion/Bounce” VM bounce.trusted.ci.jenkins.io
  • [Azure Deprecation] Convert your OS disks to Standard SSD or Premium SSD before 8 September 2028
    • Main target: agent.trusted