Infrastructure Team Meeting - July 22, 2025

[dduportal (Damien Duportal)]

2025 07 22 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @jayfranco999 (Jay Reddy)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• @biru-codeastromer (Birajit)
• @jack (Jack Zhang)

Announcements

1. Jenkins Weekly Releases

   • 2 weeks ago: 2.518 - success (despite DockerHub incident which delayed WebUI update) - You're invited to talk on Matrix
   • Last Week: 2.519 - success - You're invited to talk on Matrix
   • This Week: 2.520 - started on time - You're invited to talk on Matrix

2. Announcements:

   • LTS tomorrow! 2.516.1 (see calendar below)
   • We had a security advisory the 09 July - success - https://groups.google.com/g/jenkinsci-advisories/c/BzAzk246k38
   • Infra Roadmap
     • Jenkins Roadmap
     • Priority: finish Azure Sponsored subscription decommission
   • Someone has an apt-get issue with the Jenkins package: https://groups.google.com/g/jenkinsci-dev/c/1s1Jbg_C_tk

Upcoming Calendar

• Next Weekly: 2025-07-29 - 2.521
• Next LTS: 2025-07-23 - 2.516.1 - (Mark Waite)
  • Team will start the build so it starts earlier than Mark’s day (timezones)
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • 2025-08-08:
    • [issue to create] Digital Ocean PAT (Terraform on infra.ci) expires
    • [issue to create] VPN CRL expires
  • 2025-08-10:
    • [issue to create] Azure Credential used by release.ci to access its Keyvault expires (https://github.com/jenkins-infra/terraform-states/pull/65)

Cloud Budgets

• Azure CDF:

  • April: $3.9k (invoice)
  • May: $3.5k (invoice)
  • June: $3,474 (invoice)
  • July: $2665 (forecast at ~4k)

• Azure Sponsorship (Microsoft Credits) - Remaining: $1746 until 31 August 2025

  • April: $12.1k
  • May: $14.6k
  • June: $10.5k
  • July: $4384 => priority is to decommission this subscription as we consume ~$250 daily (6-7 days left)

• DigitalOcean - Remaining $13,448.71 until January 02, 2026

  • April: $349 (invoice)
  • May: $310 (invoice)
  • June: $252 (invoice)
  • July: $143 (forecast at $212)

• AWS:

  • CloudBees:

    • April: $532
    • May: $548
    • June: $598
    • July: $432 (forecast at $637)

  • Sponsored account ($94,370 credits lefts until 2027)

    • April: $1,977
    • May: $2,96
    • June: $367.25
    • July: $3,015 (forecast at $4,5k)

• Jfrog Artifactory Usage

  • Storage: 1.22TB (slight increase of 30 Gb)
    • Logs files and new dependencies
  • Bandwidth:
    • April: 26.34 TB (22.25 Tb for ‘releases’ repository)
    • May: 23.93 TB (22.21 Tb for ‘releases’ repository)
    • June: 22.94 TB (21.44 Tb for ‘releases’ repository)
    • July: 18.36 Tb (17.44 Tb for ‘releases’ repository) - forecast at 27 Tb
      • infra.ci.jenkins.io has jobs which download things from Artifactory. From the VM agents.
        • @en3hD3iMRx6_6IXLNY0Rag realizes it might be pipeline-step-generator and/or backend-extension-indexer (might need caches)

Notes

• Done:

  • Keep platform sane and maintainable:
    • [ci.jenkins.io] migrate from Azure (sponsored) subscription to AWS (sponsored) account
    • [trusted.ci.jenkins.io] RPU Artifactory API token expires the 2025-07-23
    • New Mirror in Germany
    • Create a “private report” web service for monitoring
  • Keep platform up to date:
    • Maven 3.9.11 upgrade campaign
    • JDK25 Early Integration
  • Support:
    • Update center is not showing most recent plugin releases
    • Container downloads fail on ci.jenkins.io with ‘failed to solve: short read: expected 32 bytes but got 0: unexpected EOF’
      • GC issue reopened
    • debian mirror not working (Redirection from https to http)
    • Verify jenkins.io Domain for JetBrains sponsorship
    • Need permission to update the branch protection rules
    • Verify Maven Central namespace for Jenkins libraries
    • [ci.jenkins.io] Jenkins jobs are failing on sync PR while buiding plugins

• Closed as not planned:

  • [ci.jenkins.io] Implement inbound protocol for EC2 agents

• Work in Progress:

  • 2025 Cloud Usage: ensure that we can run until end of year
  • Support:
    • [ci.jenkins.io] diagnose and eventually fix the EC2 Windows VM performances and build failures/retries
  • Keep platform sane and maintainable:
    • Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)
      • PR ready to review
      • Then: datadog integration
    • Tombstone Puppet (and replace it by something else)
      • Jack asks for a list of “components” from the current Puppet (roles/profiles) so we can map them to ansible roles (or others)
      • Discussion about Ansible Tower => looks like too heavy. We rather switch to “event → Ansible run” in push mode
      • Our Puppet vagrant should be replaced by Molecule
      • Question about repositories and their layouts (compared to current “all in one” Puppet repo)
        • Ansible needs separated repositories as per Jack
    • OSUOSL: decrease mirror dependency
      • Delayed (Azure decommission)
  • Support
    • Admin access for Jenkins GSoC org admins to GSoC SiG Gitter channel
      • To be delayed to August (back to triage)
    • Github Copilot organisation space
      • Nothing changed
  • Keep platform up to date:
    • Update Jira LTS from 9.12.x to 10.3.x

• Issues staying in backlog/triage:

  • private docker image registry for staging core security releases
  • GSoC 2025 - Plugin Modernizer improvement - New repo for metadata-plugin-modernizer
  • [Azure]: credentials less Service Principal
  • Tracking Issue for Groovy Script Conversion in RPU
  • Artifactory Not Caching Google’s Maven Repository
  • Migrate census.jenkins.io VM from AWS CloudBees to DigitalOcean
  • Chinese jenkins site incorrect site redirection
  • [Azure] Migrate (e.g. re-create) AKS clusters publick8s and privatek8s with modern settings (private API, Azure Linux, NAT outbound)
  • [cert.ci/trusted.ci/private.vpn] Default outbound access for VMs in Azure will be retired
  • [private.vpn.jenkins.io] Azure deprecates Public IPs of type “Basic” the 30 September 2025
  • Add a real-world job to weekly.ci.jenkins.io
  • Move collection of stats out from Kohsuke’s home
  • Support [skip ci] on default branch
  • Create build for jenkinsci/winp on release ci server
  • [Update Center] HTTP/404 on /current/updates/*.json* links
  • dnf5 update fails with gpgcheck=1
  • Add monitoring for CD secrets updates
  • [INFRA-3046] Monitor Jenkins mirrors Age

• Issues added to the next milestone:

  • JDK patch upgrade campaign (July 2025)
  • [ci.jenkins.io] Monitor and Garbage collect data volume of the DockerHub registry mirror and EC pull through cache
  • [trusted.ci.jenkins.io] migrate VM ephemeral agents from Azure (sponsored) subscription to Azure (CDF) subscription
  • [cert.ci.jenkins.io] migrate VM ephemeral agents from Azure (sponsored) subscription to Azure (CDF) subscription
  • [infra.ci.jenkins.io] migrate VM ephemeral agents from Azure (sponsored) subscription to Azure (CDF) subscription
  • [privatek8s] migrate cluster privatek8s from Azure (sponsored) subscription to Azure (CDF) subscription