Infrastructure Team Meeting - June 03, 2025

dduportal · June 7, 2025, 12:54pm

Attendees

@dduportal (Damien Duportal)
@jayfranco999 (Jay Reddy)
@smerle33 (Stéphane Merle)
@poddingue (Bruno Verachten)
@kmartens27 (Kevin Martens)

Announcements

Jenkins Weekly Releases
- This Week: 2.512 released successfully - You're invited to talk on Matrix
- This Week: 2.513 started on time - You're invited to talk on Matrix
Announcements:
- Most of the infra team is in low bandwitdh this week (Damien off Friday, corporate event for Mark, Stephane and Jay, Monday is banking day in France at least)
- LTS 2.504.2 released last week, no issues

Upcoming Calendar

Next Weekly: 2025-06-10, 2.514, should be the first built with JDK21
Next LTS: 2025-06-25, 2.504.3, Philipp Glanz is the release lead, ref. Events
- Not sure yet (no backport as for today)
Next Security Release as per jenkinsci-advisories: N.A.
Upcoming credentials expirations (~3 weeks):
- 2025-06-14:
  - (Issue to be done) Azure SP used by infra.ci to spin up Azure VM agents: Azure AD Application password for Azure VM agents in `infra.ci.jenkins.io` expires on `2025-06-14T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #1049 · jenkins-infra/azure · GitHub
    - Should be converted to credential-less workload identity (ref. Jenkins Controllers in Azure Kubernetes: use workload identity management to allow managing Azure VM agents without credentials · Issue #4651 · jenkins-infra/helpdesk · GitHub)
    - @smerle + @dduportal
- 2025-06-15: Azure SP used to deploy websites to production in infra.ci (into file shares):
- 2025-06-16: Azure SP used for Terraform backends for projects (issue to create, with all other terraform state below):
  - AWS (CloudBees) - https://github.com/jenkins-infra/terraform-states/pull/56
- 2025-06-17: Azure SP used for Terraform backends (and some API tokens) for projects (issue to create):
  - AWS (Sponsored) - https://github.com/jenkins-infra/terraform-states/pull/57
  - Azure - https://github.com/jenkins-infra/terraform-states/pull/58
  - Azure Net - https://github.com/jenkins-infra/terraform-states/pull/59
  - Cloudflare - https://github.com/jenkins-infra/terraform-states/pull/60
    - @smerle to ensure @dduportal is not bus factor
  - Datadog - https://github.com/jenkins-infra/terraform-states/pull/61
  - Digital Ocean - https://github.com/jenkins-infra/terraform-states/pull/62
  - Fastly - https://github.com/jenkins-infra/terraform-states/pull/63
Next major event: N.A.

Cloud Budgets

Azure CDF:
- March: $4,3k (invoice)
- April: $3,9k (invoice)
- May: ~$3,166 (raw evaluation with CDF due to cost data hidden by Microsoft until billing issues are settled)
- June: $222 (forecast at ~$3.1k)
Azure Sponsorship (Microsoft Credits) - Remaining: $15,747 until 31 August 2025
- March: $4,276
- April: $12.1k
- May: $14,6k
- June: $890 => last month with this subscription
DigitalOcean - Remaining $13,832 until January 02, 2026
- March: $272 (invoice)
- April: $349 (invoice)
- May: $310 (invoice)
- June: $12 (forecast at $360)
AWS:
- CloudBees:
  - March: $551
  - April: $532
  - May: $548
  - June: $28 (forecast at $536)
- Sponsored account (~$34,747 credits lefts until 01/31/2027)
  - March: $14,649
  - April: $1,977
  - May: $2,96
  - June: $0,17 (forecast at $3)
Jfrog Artifactory Usage
- Storage: 1.32TB
- Bandwidth:
  - March: 35.25 TB (better than expected)
  - April: 26.34 TB (22.25 Tb for ‘releases’ repository)
  - May: 23.93 Tb (22.21 Tb for ‘releases’ repository)
  - June: 2.16Tb (2.04 Tb for ‘releases’ repository, forecast at ~20 TB)

Notes

Done:
- Support:
  - Request of plugin-health application logs
  - https://weekly.ci.jenkins.io/ header is broken
- Keep infrastructure up to date:
  - [infra.ci.jenkins.io] NPM token expires the 04 June 2025
  - Update ci.jenkins.io, trusted.ci, cert.ci and release.ci to latest LTS version 2.504.2
  - [Upgrade Campaign] Bump Cloudflare Terraform provider to 5.x
    - Switched to account API tokens (instead of user API tokens)
    - R2 access tokens are now managed publicly
    - @smerle should rottate tokens next time to ensure @dduportal is not bus factor
  - [infra.ci.jenkins.io] Netlify token expires 04 June 2025
  - [infra-reports] Sunset the “plugin-migration” report
  - packer-images: use api.adoptium.net for updatecli and installation of JDKs
    - Less code divergence between version tracking, ubuntu installation and windows installation
    - Less risk of trying to bump a version while its platform is nto available
    - Paves the way for JDK25
    - Thanks @gounthar for the help!
Closed as not planned:
- Switch default JDK to 21 for pipeline libraries
Work in Progress:
- Support:
  - Artifact caching proxy HTTP 403 fails many jobs on ci.jenkins.io
    - Next step: patch ACP to answer HTTP/404 when all backends answers HTTP/404 (or HTTP/403)
  - sg.mirror.servanamanaged.com mirror returns 404
    - Mirror disabled to not have user-facing errors in Asia/Oceania
      - TWDS and OSSPlanet are covering for this areas (we also have 2 mirrors in Japan if need be)
      - In 7 days, without answer from Servana, we delete the mirror
  - JDK25 integration
    - All requirements have been done (updatecli, etc.)
    - Waiting for JDK21 default everywhere
    - Need to add in packer-images, but also in tools (ci.jio, cert.ci and trusted.ci)
  - Restore jenkinsci/jenkins build stability
- Keep infrastructure up to date:
  - Update Jira LTS from 9.12.x to 10.3.x
    - No news from LF (yet?), Mark still on it
- Keep infrastructure healthy:
  - Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)
    - Base logic is done (draft PR opened) on the library (generates JSON on the agent with a shell and try to publish it)
      - Naming to update
      - Blocked by missing credential for publishing on certain jobs
    - Security questions: we might need to set up a new service to manage credential properly and update publishReports() accordingly
  - OSUOSL: decrease mirror dependency
    - Was delayed
    - New issue for mirror provider: OSUOSL only stored 1 year of artifacts (~250Gb) while archives.jenkins.io stores all (~600Gb)
      - We need a “mirror maintainer” public documentation with this warning added (they have to limit the age of artifact to 1 year)
    - Next step: sync scripts on pkg.
  - jgit in Artifactory proxies all of Eclipse and jgit-cache contains 200GB
    - Nothing done, need help from Darin and Mark
  - Jenkins Controllers in Azure Kubernetes: use workload identity management to allow managing Azure VM agents without credentials
    - Team effort which outcome was an exhaustive list of changes to apply
    - Wip on Terraform Azure, figthing with modules to not break existing controllers
- Use JDK21 Platform-wide
  - Switch agent (java home/Path) to JDK21 default
    - Jenkins Weekl 2.514 should be built with JDK21 instead of JDK21
    - Issue updated (cleanup and removing misunderstanding) by @smerle
    - WiP on Draft PRs sent for agents in ci.jio and other locations
- 2025 Cloud Usage: ensure that we can run until end of year
  - Back to backlog as we’ve secured funds for end of year until we open remaining issues from Azure advisor
Issues added to the next milestone:

Topic	Replies	Views
Infrastructure Team Meeting - June 24, 2025 Infrastructure meeting , sig-infra	22	June 26, 2025
Infrastructure Team Meeting - June 17, 2025 Infrastructure meeting , sig-infra	16	June 19, 2025
Infrastructure Team Meeting - June 10, 2025 Infrastructure meeting , sig-infra	16	June 16, 2025
Infrastructure Team Meeting - May 27, 2025 Infrastructure meeting , sig-infra	23	June 3, 2025
Infrastructure Team Meeting - July 01, 2025 Infrastructure meeting , sig-infra	26	July 8, 2025

Infrastructure Team Meeting - June 03, 2025

Attendees

Announcements

Upcoming Calendar

Cloud Budgets

Notes

Related topics