Infrastructure Team Meeting - July 01, 2025

[dduportal (Damien Duportal)]

2025 07 01 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• @zhan9san (Jack Zhang)
• @biru-codeastromer (Birajit Saikia)
• @kmartens27 (Kevin Martens)

Announcements

1. Jenkins Weekly Releases
   • Last Week: 2.516 - delayed packaging due to Windows Git 2.50.0 issue - You're invited to talk on Matrix
   • This Week: 2.517 - started on time - You're invited to talk on Matrix
     • We’re back to Git 2.49 for Windows packaging
2. Announcements:
   • Jenkins LTS 2.504.3 succesfully released the 24 June 2025
   • DockerHub Automatic email about Open Source subscription canceled
     • https://groups.google.com/g/jenkins-infra-team/c/HRQfC9Mleqc/m/MYBFtI_MAAAJ
   • Java Patch Day the 15 July
   • Infra roadmap
     • Proposed update in feat(roadmap) update jenkins-infra tasks for June 2025 by dduportal · Pull Request #8209 · jenkins-infra/jenkins.io · GitHub
     • Preview: https://deploy-preview-8209--jenkins-io-site-pr.netlify.app/project/roadmap/
   • New meeting time to discuss to cover Asia (Jack, Jay, Birajit): next week to discuss
     • Jack is ok with current meeting time

Upcoming Calendar

• Next Weekly: 2025-07-08 - 2.518

• Next LTS: (?)

• Next Security Release as per jenkinsci-advisories: N.A.

• Upcoming credentials expirations (~3 weeks):

  • 2025-07-06:
    • Azure SP used for updatecli jobs in infra.ci: Azure AD Application password for updatecli in `infra.ci.jenkins.io` expires on `2025-07-06T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #1063 · jenkins-infra/azure · GitHub
      • [azure] renewal of differents Service Principal Passwords · Issue #4713 · jenkins-infra/helpdesk · GitHub
    • Azure SP used to deploy websites to production from trusted.ci (into file shares)
      • www.jenkins.io: Azure File Share Principal `www.jenkins.io` on `trusted.ci.jenkins.io` expires on `2025-07-06T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #1062 · jenkins-infra/azure · GitHub
      • javadoc.jenkins.io: Azure File Share Principal `javadoc.jenkins.io` on `trusted.ci.jenkins.io` expires on `2025-07-06T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #1061 · jenkins-infra/azure · GitHub
      • good candidate for moving to a credential-less by using ephemeral VM agent assigned identity (trusted.ci is NOT multi tenant)
      • [azure] renewal of differents Service Principal Passwords · Issue #4713 · jenkins-infra/helpdesk · GitHub
    • Azure SP used to deploy websites to production from infra.ci (into file shares):
      • docs.jenkins.io: New end date for `docs.jenkins.io` File Share service principal writer on `infra.ci.jenkins.io` (current: "2025-07-06T00:00:00Z") by jenkins-infra-updatecli[bot] · Pull Request #1059 · jenkins-infra/azure · GitHub
      • [azure] renewal of differents Service Principal Passwords · Issue #4713 · jenkins-infra/helpdesk · GitHub
    • Azure SP used by geoip Jobs in cluster publick8s to update geolocation DB for mirrorbits: Azure File Share Principal `geoip` on `publick8s` expires on `2025-07-06T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #1060 · jenkins-infra/azure · GitHub
      • good candidate for moving to a credential-less workload identity. Jenkins Azure plugin do not support it (yet) as per Jenkins Controllers in Azure Kubernetes: use workload identity management to allow managing Azure VM agents without credentials · Issue #4651 · jenkins-infra/helpdesk · GitHub BUT this one is a direct pod → Azure File Storage thing so it should work well.
      • [azure] renewal of differents Service Principal Passwords · Issue #4713 · jenkins-infra/helpdesk · GitHub

• Next major event: N.A.

Cloud Budgets

• Azure CDF:

  • April: $3.9k (invoice)
  • May: $3.5k (invoice)
  • June: 3,474 (+100) - waiting for invoice of $3,5k
  • July: too soon to evaluate or forecast

• Azure Sponsorship (Microsoft Credits) - Remaining: $6040 until 31 August 2025

  • April: $12.1k
  • May: $14.6k
  • June: $10.5k
  • July: too soon to evaluate or forecast

• DigitalOcean - Remaining $13,591.09 until January 02, 2026

  • April: $349 (invoice)
  • May: $310 (invoice)
  • June: $252
  • July: too soon to evaluate or forecast

• AWS:

  • CloudBees:

    • April: $532
    • May: $548
    • June: $598
    • July: too soon to evaluate or forecast

  • Sponsored account ($94,747 credits lefts until 2027)

    • April: $1,977
    • May: $2,96
    • June: $367.25
    • July: too soon to evaluate or forecast

• Jfrog Artifactory Usage

  • Storage: 1.18TB (slight decrease thanks to Darin’s work on jgit cache)
  • Bandwidth:
    • April: 26.34 TB (22.25 Tb for ‘releases’ repository)
    • May: 23.93 TB (22.21 Tb for ‘releases’ repository)
    • June: 22.94.7TB (21.44TB Tb for ‘releases’ repository)
    • July: too soon to evaluate or forecast

Notes

• Done:

  • ci.jenkins.io from Azure to AWS:
    • [Packer-Images] enable AWS as a destination for packer-images templates
      • Goss test harness for Windows because they are really unstable (false positive and passing tests when they should not)
  • Keep platform up to date:
    • Update ci.jenkins.io, trusted.ci, cert.ci and release.ci to latest LTS version 2.504.3
  • Keep platform sane and maintainable:
    • [infra.ci.jenkins.io] migrate container agents (cluster infracijenkinsio-agents-1) from Azure (sponsored) subscription to Azure (CDF) subscription
      • A few infra-reports long running builds are aborted due to spot reclaimed instance
    • [Jenkins Agents] Clean up deprecated JNLP arguments

• Work in Progress:

  • 2025 Cloud Usage: ensure that we can run until end of year

    • [ci.jenkins.io] migrate from Azure (sponsored) subscription to AWS (sponsored) account
      • Stephane worked on the VM (Puppet code), almost to the first provisioning
      • We have the EKS cluster up and running
      • Then tests, and effective migration

  • Stats

    • [stats.jenkins.io/infra-statistics] Move “data for the usage stats site” generation (from anonymized data) out from Andrew machine)
      • Delayed: focused on AWS migration. Moving to mid-July
      • Note: there also gaps in the last months stats: need diagnose

  • Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)

    • Create a “private report” web service for monitoring
      • Only release.ci container agents are in “todo”
    • Jay succesfully finished his pipeline library PR. Need to review it and integrate it.
    • Next step: Jay will investigate the datadog HTTP monitor

  • Support:

    • Admin access for Jenkins GSoC org admins to GSoC SiG Gitter channel
      • need to contact Element for takeover. Jenkins Board?
    • Github Copilot organisation space
      • jenkinsci admin hands. No infra task related.

  • Keep platform up to date:

    • Update Jira LTS from 9.12.x to 10.3.x
      • No news from LF. Independance Day holidays?
    • JDK25 integration
      • Delayed: need to resume work on Jenkins tools.

  • Keep platform sane and maintainable:

    • [azure] renewal of differents Service Principal Passwords
      • Let’s start!
    • OSUOSL: decrease mirror dependency
      • Delayed (until after the AS ci.jio migration)
    • jgit in Artifactory proxies all of Eclipse and jgit-cache contains 200GB
      • In Darin’s hands.

• Issues staying in backlog/triage:

  • Tracking Issue for Groovy Script Conversion in RPU
  • Tombstone Puppet (and replace it by something else)
  • Artifactory Not Caching Google’s Maven Repository
  • Migrate census.jenkins.io VM from AWS CloudBees to DigitalOcean
  • [trusted.ci.jenkins.io] migrate VM ephemeral agents from Azure (sponsored) subscription to Azure (CDF) subscription
  • [cert.ci.jenkins.io] migrate VM ephemeral agents from Azure (sponsored) subscription to Azure (CDF) subscription
  • [infra.ci.jenkins.io] migrate VM ephemeral agents from Azure (sponsored) subscription to Azure (CDF) subscription
  • [privatek8s] migrate cluster privatek8s from Azure (sponsored) subscription to Azure (CDF) subscription
  • Chinese jenkins site incorrect site redirection
  • [Azure] Migrate (e.g. re-create) AKS clusters publick8s and privatek8s with modern settings (private API, Azure Linux, NAT outbound)
  • [cert.ci/trusted.ci/private.vpn] Default outbound access for VMs in Azure will be retired
  • [private.vpn.jenkins.io] Azure deprecates Public IPs of type “Basic” the 30 September 2025
  • Add a real-world job to weekly.ci.jenkins.io
  • Move collection of stats out from Kohsuke’s home
  • Support [skip ci] on default branch
  • Create build for jenkinsci/winp on release ci server
  • [Update Center] HTTP/404 on /current/updates/*.json* links
  • dnf5 update fails with gpgcheck=1
  • Add monitoring for CD secrets updates
  • [INFRA-3046] Monitor Jenkins mirrors Age

• Issues added to the next milestone:

  • Renew Docker Open Source Program sponsorship for 2025-2026
  • New Mirror in Germany